2024-01-19T06:06:52.746 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-19T06:06:52.746 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124 --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124/orig.config.yaml 2024-01-19T06:06:52.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T06:06:52.897 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124 branch: wip-yuri8-testing-2024-01-18-0823 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7521124' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-01-18-0823 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0edf41a622739843d6b978b179ff3227b476dd9d ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0edf41a622739843d6b978b179ff3227b476dd9d kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-01-18-0823 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6631 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2024-01-18-0823 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0edf41a622739843d6b978b179ff3227b476dd9d targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9tsHnbIlHfuRN24Az3gKwr6oYVI2AMLDFh8/ngO0GhHek8AUVOOJoIjszuTsRS09+3q3OyocQXSNQa5BFHzpc= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJVfPBi7HetMwVhieXrb1/31pCfddChakSk8rx6raRJkUU5FqQajV20To/4zLM/P1zNPT0yK2+Vr/DVkpjt3zSo= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-18_21:19:14 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-19T06:06:52.898 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa; will attempt to use it 2024-01-19T06:06:52.898 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks 2024-01-19T06:06:52.899 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-19T06:06:52.900 INFO:teuthology.task.internal:Checking packages... 2024-01-19T06:06:52.923 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0edf41a622739843d6b978b179ff3227b476dd9d' 2024-01-19T06:06:52.923 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:06:52.923 INFO:teuthology.packaging:ref: None 2024-01-19T06:06:52.923 INFO:teuthology.packaging:tag: None 2024-01-19T06:06:52.923 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-01-18-0823 2024-01-19T06:06:52.924 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:06:52.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-01-18-0823 2024-01-19T06:06:53.088 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-766.g0edf41a6 2024-01-19T06:06:53.090 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-19T06:06:53.095 INFO:teuthology.task.internal:no buildpackages task found 2024-01-19T06:06:53.096 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-19T06:06:53.113 INFO:teuthology.task.internal:Saving configuration 2024-01-19T06:06:53.135 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-19T06:06:53.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T06:06:53.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 06:01:29.156475', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9tsHnbIlHfuRN24Az3gKwr6oYVI2AMLDFh8/ngO0GhHek8AUVOOJoIjszuTsRS09+3q3OyocQXSNQa5BFHzpc='} 2024-01-19T06:06:53.186 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 06:01:29.158719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJVfPBi7HetMwVhieXrb1/31pCfddChakSk8rx6raRJkUU5FqQajV20To/4zLM/P1zNPT0yK2+Vr/DVkpjt3zSo='} 2024-01-19T06:06:53.186 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-19T06:06:53.192 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-19T06:06:53.192 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-19T06:06:53.192 INFO:teuthology.run_tasks:Running task console_log... 2024-01-19T06:06:53.260 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1210ce4160>, signals=[15]) 2024-01-19T06:06:53.260 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-19T06:06:53.266 INFO:teuthology.task.internal:Opening connections... 2024-01-19T06:06:53.267 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:06:53.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:06:53.358 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-01-19T06:06:53.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:06:53.448 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-19T06:06:53.454 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-01-19T06:06:53.472 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-01-19T06:06:53.472 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-01-19T06:06:53.529 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-01-19T06:06:53.530 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T06:06:53.530 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-01-19T06:06:53.530 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T06:06:53.530 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T06:06:53.530 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T06:06:53.530 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-01-19T06:06:53.549 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-01-19T06:06:53.565 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-01-19T06:06:53.566 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-01-19T06:06:53.625 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-01-19T06:06:53.626 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T06:06:53.626 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-01-19T06:06:53.626 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T06:06:53.626 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T06:06:53.626 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T06:06:53.626 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-01-19T06:06:53.649 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-19T06:06:53.657 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-19T06:06:53.663 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-19T06:06:53.663 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-01-19T06:06:53.665 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-01-19T06:06:53.683 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-19T06:06:53.688 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-19T06:06:53.689 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-01-19T06:06:53.721 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-01-19T06:06:53.812 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-19T06:06:53.866 INFO:teuthology.run_tasks:Running task kernel... 2024-01-19T06:06:53.881 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-19T06:06:53.881 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-19T06:06:53.881 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T06:06:53.882 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T06:06:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:06:53.897 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-01-19T06:06:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:06:53.898 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-19T06:06:53.954 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-01-19T06:06:53.954 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-01-19T06:06:53.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-01-19T06:06:53.955 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-01-19T06:06:53.955 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-01-19T06:06:53.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-01-19T06:06:56.051 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2024-01-19T06:06:56.124 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2024-01-19T06:06:57.401 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-01-19T06:06:57.475 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-01-19T06:06:58.559 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-01-19T06:07:02.315 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-01-19T06:07:04.858 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-01-19T06:07:07.417 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-01-19T06:07:09.375 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 5.9 MB 00:01 2024-01-19T06:07:10.925 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-01-19T06:07:12.926 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.9 MB 00:02 2024-01-19T06:07:13.019 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-01-19T06:07:15.293 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2024-01-19T06:07:16.211 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-01-19T06:07:17.802 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-01-19T06:07:19.941 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T06:07:19.942 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T06:07:20.045 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:07:20.053 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:07:20.054 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-01-19T06:07:20.055 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:07:20.055 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-01-19T06:07:20.055 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 122 M 2024-01-19T06:07:20.055 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:07:20.419 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-01-19T06:07:21.036 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-01-19T06:07:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-01-19T06:07:21.728 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-01-19T06:07:22.062 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-408.el9.x86_6 20 MB/s | 33 MB 00:01 2024-01-19T06:07:22.065 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:07:22.065 INFO:teuthology.orchestra.run.smithi134.stdout:Total 49 MB/s | 99 MB 00:02 2024-01-19T06:07:22.066 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:07:22.425 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:07:22.425 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:07:23.218 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T06:07:23.218 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T06:07:23.296 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:07:23.303 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 99 M 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 122 M 2024-01-19T06:07:23.304 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:07:23.478 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:07:23.478 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:07:23.654 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-01-19T06:07:24.229 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-01-19T06:07:24.846 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 25 MB/s | 39 MB 00:01 2024-01-19T06:07:25.230 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-core-5.14.0-408.el9.x86_6 21 MB/s | 33 MB 00:01 2024-01-19T06:07:25.233 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:07:25.233 INFO:teuthology.orchestra.run.smithi150.stdout:Total 51 MB/s | 99 MB 00:01 2024-01-19T06:07:25.233 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:07:25.593 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:07:25.594 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:07:25.884 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:07:26.659 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:07:26.659 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:07:26.674 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-19T06:07:26.811 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:07:29.062 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:07:29.587 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:07:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:07:29.826 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-19T06:07:29.972 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:07:32.810 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:07:32.999 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:07:34.429 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:07:34.626 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:07:37.735 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:07:37.927 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:07:39.339 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:07:42.676 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:08.832 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T06:08:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:08:08.902 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:09.344 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:09.344 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-19T06:08:09.344 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:08:09.344 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:08:09.762 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:08:10.268 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T06:08:10.401 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:10.401 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T06:08:10.401 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-19T06:08:10.401 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T06:08:10.401 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T06:08:10.402 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-19T06:08:10.402 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T06:08:10.402 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T06:08:10.510 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:10.510 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T06:08:10.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-01-19T06:08:10.554 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T06:08:10.555 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:08:10.556 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T06:08:10.557 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T06:08:10.557 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-408.el9.x86_64 2024-01-19T06:08:10.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T06:08:11.112 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-01-19T06:08:11.627 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:11.627 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T06:08:11.627 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:08:11.697 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:12.206 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:12.206 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-19T06:08:12.206 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T06:08:12.206 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T06:08:12.535 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T06:08:12.555 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-01-19T06:08:12.558 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-01-19T06:08:12.590 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.5OG7euXjZP 2024-01-19T06:08:12.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5OG7euXjZP 2024-01-19T06:08:12.650 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:08:12.651 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:08:12.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.5OG7euXjZP 2024-01-19T06:08:12.825 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.5OG7euXjZP is 6KB 2024-01-19T06:08:12.874 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.5OG7euXjZP 2024-01-19T06:08:12.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T06:08:12.952 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T06:08:12.953 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T06:08:12.953 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T06:08:12.953 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-19T06:08:12.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-19T06:08:13.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-01-19T06:08:13.181 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T06:08:13.320 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:13.320 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T06:08:13.320 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-19T06:08:13.320 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T06:08:13.320 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T06:08:13.320 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-19T06:08:13.320 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T06:08:13.320 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T06:08:13.428 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T06:08:13.428 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T06:08:13.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-01-19T06:08:13.465 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-01-19T06:08:13.465 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-01-19T06:08:13.466 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:08:13.467 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T06:08:13.468 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T06:08:13.468 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-408.el9.x86_64 2024-01-19T06:08:13.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T06:08:14.085 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-01-19T06:08:15.472 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T06:08:15.514 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-01-19T06:08:15.516 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-01-19T06:08:15.532 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.83GnrXUxvl 2024-01-19T06:08:15.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.83GnrXUxvl 2024-01-19T06:08:15.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.83GnrXUxvl 2024-01-19T06:08:15.740 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.83GnrXUxvl is 6KB 2024-01-19T06:08:15.788 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.83GnrXUxvl 2024-01-19T06:08:15.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T06:08:15.866 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T06:08:15.867 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T06:08:15.867 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T06:08:15.867 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-19T06:08:15.868 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-19T06:08:16.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-01-19T06:08:43.145 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-19T06:08:43.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-19T06:08:43.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:08:46.028 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-19T06:08:46.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:08:46.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:09:04.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T06:09:13.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:09:13.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:09:16.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T06:09:28.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:09:28.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:09:43.163 DEBUG:teuthology.orchestra.remote:timed out 2024-01-19T06:09:52.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-19T06:09:52.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:09:55.697 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-19T06:09:56.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-19T06:09:56.184 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-408.el9.x86_64"... 2024-01-19T06:09:56.184 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-19T06:09:56.198 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-408.el9.x86_64 2024-01-19T06:09:56.198 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-19T06:09:56.198 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T06:09:56.198 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T06:09:57.198 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-19T06:09:57.199 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T06:09:57.277 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-01-19T06:09:57.330 DEBUG:teuthology.parallel:result is None 2024-01-19T06:10:28.643 DEBUG:teuthology.orchestra.remote:timed out 2024-01-19T06:10:43.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:10:43.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:10:44.056 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-01-19T06:10:44.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:10:44.446 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-408.el9.x86_64"... 2024-01-19T06:10:44.446 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-01-19T06:10:44.461 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-408.el9.x86_64 2024-01-19T06:10:44.462 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-19T06:10:44.462 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T06:10:44.462 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T06:10:45.463 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-19T06:10:45.463 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T06:10:45.554 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-01-19T06:10:45.585 DEBUG:teuthology.parallel:result is None 2024-01-19T06:10:45.585 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-19T06:10:45.593 INFO:teuthology.task.internal:Creating test directory... 2024-01-19T06:10:45.593 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T06:10:45.598 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T06:10:45.615 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-19T06:10:45.622 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-19T06:10:45.691 INFO:teuthology.task.internal:Creating archive directory... 2024-01-19T06:10:45.691 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T06:10:45.694 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T06:10:45.726 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-19T06:10:45.734 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-19T06:10:45.734 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T06:10:45.762 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T06:10:45.800 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T06:10:45.814 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T06:10:45.814 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T06:10:45.827 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T06:10:45.829 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-19T06:10:45.835 INFO:teuthology.task.internal:Configuring sudo... 2024-01-19T06:10:45.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T06:10:45.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T06:10:45.885 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-19T06:10:45.894 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-19T06:10:45.894 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T06:10:45.908 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T06:10:45.941 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T06:10:46.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T06:10:46.053 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T06:10:46.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T06:10:46.178 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:10:46.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T06:10:46.246 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T06:10:46.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T06:10:46.352 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T06:10:46.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T06:10:46.439 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:10:46.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T06:10:46.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-01-19T06:10:46.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-01-19T06:10:46.576 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T06:10:46.601 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T06:10:46.956 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-19T06:10:46.964 INFO:teuthology.task.internal:Starting timer... 2024-01-19T06:10:46.964 INFO:teuthology.run_tasks:Running task pcp... 2024-01-19T06:10:46.976 INFO:teuthology.run_tasks:Running task selinux... 2024-01-19T06:10:46.984 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-19T06:10:46.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-01-19T06:10:47.052 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-01-19T06:10:47.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-01-19T06:10:47.095 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-01-19T06:10:47.097 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-19T06:10:47.097 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-01-19T06:10:47.124 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-01-19T06:10:47.124 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-01-19T06:10:47.160 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-01-19T06:10:47.160 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2024-01-19T06:10:47.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T06:10:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:10:47.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T06:10:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:10:47.230 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-19T06:10:47.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-01-19T06:10:47.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-01-19T06:10:47.304 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-19T06:10:47.313 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-19T06:10:47.330 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-19T06:10:47.331 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-19T06:14:43.627 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-01-19T06:14:43.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-19T06:14:43.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:14:43.716 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-19T06:14:43.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-19T06:14:43.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:14:43.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T06:14:43.888 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-01-19T06:14:43.966 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T06:14:43.966 INFO:teuthology.run_tasks:Running task clock... 2024-01-19T06:14:43.976 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-19T06:14:43.976 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T06:14:43.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T06:14:43.979 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T06:14:43.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T06:14:44.008 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T06:14:44.025 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T06:14:44.037 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T06:14:44.053 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T06:14:44.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-01-19T06:14:44.068 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-01-19T06:14:44.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-01-19T06:14:44.084 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T06:14:44.095 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-01-19T06:14:44.100 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T06:14:44.112 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T06:14:44.128 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T06:14:44.151 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.181 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-01-19T06:14:44.184 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T06:14:44.184 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-01-19T06:14:44.185 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.185 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.185 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.185 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T06:14:44.185 INFO:teuthology.run_tasks:Running task install... 2024-01-19T06:14:44.192 DEBUG:teuthology.task.install:project ceph 2024-01-19T06:14:44.193 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}} 2024-01-19T06:14:44.193 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-19T06:14:44.193 INFO:teuthology.task.install:Using flavor: default 2024-01-19T06:14:44.198 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-19T06:14:44.198 INFO:teuthology.task.install:extra packages: [] 2024-01-19T06:14:44.199 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-19T06:14:44.199 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:14:44.199 INFO:teuthology.packaging:ref: None 2024-01-19T06:14:44.199 INFO:teuthology.packaging:tag: None 2024-01-19T06:14:44.199 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:14:44.199 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:14:44.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T06:14:44.201 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-19T06:14:44.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:14:44.201 INFO:teuthology.packaging:ref: None 2024-01-19T06:14:44.201 INFO:teuthology.packaging:tag: None 2024-01-19T06:14:44.201 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:14:44.201 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:14:44.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T06:14:44.378 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/ 2024-01-19T06:14:44.378 INFO:teuthology.task.install.rpm:Package version is 17.2.7-268.g6441305d 2024-01-19T06:14:44.383 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/ 2024-01-19T06:14:44.383 INFO:teuthology.task.install.rpm:Package version is 17.2.7-268.g6441305d 2024-01-19T06:14:44.486 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T06:14:44.486 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:14:44.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T06:14:44.499 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T06:14:44.499 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:14:44.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T06:14:44.514 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T06:14:44.514 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:14:44.514 INFO:teuthology.packaging:ref: None 2024-01-19T06:14:44.514 INFO:teuthology.packaging:tag: None 2024-01-19T06:14:44.514 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:14:44.514 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:14:44.515 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T06:14:44.527 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T06:14:44.527 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:14:44.527 INFO:teuthology.packaging:ref: None 2024-01-19T06:14:44.527 INFO:teuthology.packaging:tag: None 2024-01-19T06:14:44.527 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:14:44.527 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:14:44.527 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T06:14:44.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T06:14:44.594 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T06:14:44.661 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T06:14:44.674 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T06:14:44.727 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-01-19T06:14:44.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-01-19T06:14:44.741 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-01-19T06:14:44.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-01-19T06:14:44.963 INFO:teuthology.orchestra.run.smithi134.stdout:73 files removed 2024-01-19T06:14:44.980 INFO:teuthology.orchestra.run.smithi150.stdout:73 files removed 2024-01-19T06:14:44.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-01-19T06:14:45.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2024-01-19T06:14:45.667 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 215 kB/s | 77 kB 00:00 2024-01-19T06:14:45.670 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 225 kB/s | 77 kB 00:00 2024-01-19T06:14:46.010 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 31 kB/s | 9.1 kB 00:00 2024-01-19T06:14:46.037 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2024-01-19T06:14:46.335 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.2 kB/s | 1.8 kB 00:00 2024-01-19T06:14:46.379 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-01-19T06:14:47.046 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.4 MB 00:00 2024-01-19T06:14:47.056 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-19T06:14:48.671 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-01-19T06:14:48.723 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-01-19T06:14:49.703 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-01-19T06:14:52.657 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-01-19T06:14:54.496 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-01-19T06:14:55.777 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 19 MB 00:02 2024-01-19T06:14:59.905 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.9 MB 00:01 2024-01-19T06:15:01.555 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-01-19T06:15:02.017 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 5.9 MB 00:02 2024-01-19T06:15:03.634 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-01-19T06:15:03.959 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 20 MB 00:02 2024-01-19T06:15:04.636 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2024-01-19T06:15:08.740 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-19T06:15:09.427 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-01-19T06:15:11.260 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 14 M 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.1 M 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:15:11.270 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:17.2.7-268.g6441305d.el9 ceph 5.0 M 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 23 M 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 634 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 458 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 4.9 M 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T06:15:11.271 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:17.2.7-268.g6441305d.el9 ceph 45 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 116 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:17.2.7-268.g6441305d.el9 ceph 156 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:17.2.7-268.g6441305d.el9 ceph 319 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 293 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 99 k 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout:Install 22 Packages 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-01-19T06:15:11.272 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:11.273 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 58 M 2024-01-19T06:15:11.273 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:15:11.928 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 14 M 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.1 M 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:17.2.7-268.g6441305d.el9 ceph 5.0 M 2024-01-19T06:15:11.942 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 23 M 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 634 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 458 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 4.9 M 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:17.2.7-268.g6441305d.el9 ceph 45 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 116 k 2024-01-19T06:15:11.943 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:17.2.7-268.g6441305d.el9 ceph 156 k 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:17.2.7-268.g6441305d.el9 ceph 319 k 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 293 k 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 99 k 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout:Install 22 Packages 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2024-01-19T06:15:11.944 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:11.945 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 58 M 2024-01-19T06:15:11.945 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:15:12.549 INFO:teuthology.orchestra.run.smithi134.stdout:(1/24): ceph-base-17.2.7-268.g6441305d.el9.x86_ 5.4 MB/s | 5.0 MB 00:00 2024-01-19T06:15:12.633 INFO:teuthology.orchestra.run.smithi134.stdout:(2/24): ceph-selinux-17.2.7-268.g6441305d.el9.x 302 kB/s | 25 kB 00:00 2024-01-19T06:15:12.958 INFO:teuthology.orchestra.run.smithi134.stdout:(3/24): libcephfs2-17.2.7-268.g6441305d.el9.x86 1.9 MB/s | 634 kB 00:00 2024-01-19T06:15:13.125 INFO:teuthology.orchestra.run.smithi134.stdout:(4/24): libradosstriper1-17.2.7-268.g6441305d.e 2.7 MB/s | 458 kB 00:00 2024-01-19T06:15:13.709 INFO:teuthology.orchestra.run.smithi134.stdout:(5/24): ceph-common-17.2.7-268.g6441305d.el9.x8 11 MB/s | 23 MB 00:02 2024-01-19T06:15:13.767 INFO:teuthology.orchestra.run.smithi134.stdout:(6/24): python3-ceph-argparse-17.2.7-268.g64413 778 kB/s | 45 kB 00:00 2024-01-19T06:15:13.834 INFO:teuthology.orchestra.run.smithi134.stdout:(7/24): python3-ceph-common-17.2.7-268.g6441305 1.7 MB/s | 116 kB 00:00 2024-01-19T06:15:13.837 INFO:teuthology.orchestra.run.smithi150.stdout:(1/24): ceph-base-17.2.7-268.g6441305d.el9.x86_ 2.8 MB/s | 5.0 MB 00:01 2024-01-19T06:15:13.935 INFO:teuthology.orchestra.run.smithi134.stdout:(8/24): librgw2-17.2.7-268.g6441305d.el9.x86_64 6.0 MB/s | 4.9 MB 00:00 2024-01-19T06:15:13.945 INFO:teuthology.orchestra.run.smithi150.stdout:(2/24): ceph-selinux-17.2.7-268.g6441305d.el9.x 232 kB/s | 25 kB 00:00 2024-01-19T06:15:13.960 INFO:teuthology.orchestra.run.smithi134.stdout:(9/24): python3-cephfs-17.2.7-268.g6441305d.el9 1.2 MB/s | 156 kB 00:00 2024-01-19T06:15:13.994 INFO:teuthology.orchestra.run.smithi134.stdout:(10/24): python3-rados-17.2.7-268.g6441305d.el9 5.3 MB/s | 319 kB 00:00 2024-01-19T06:15:14.035 INFO:teuthology.orchestra.run.smithi134.stdout:(11/24): python3-rbd-17.2.7-268.g6441305d.el9.x 3.8 MB/s | 293 kB 00:00 2024-01-19T06:15:14.060 INFO:teuthology.orchestra.run.smithi134.stdout:(12/24): python3-rgw-17.2.7-268.g6441305d.el9.x 1.5 MB/s | 99 kB 00:00 2024-01-19T06:15:14.104 INFO:teuthology.orchestra.run.smithi150.stdout:(3/24): libcephfs2-17.2.7-268.g6441305d.el9.x86 3.9 MB/s | 634 kB 00:00 2024-01-19T06:15:14.169 INFO:teuthology.orchestra.run.smithi134.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 407 kB/s | 44 kB 00:00 2024-01-19T06:15:14.194 INFO:teuthology.orchestra.run.smithi134.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-01-19T06:15:14.245 INFO:teuthology.orchestra.run.smithi134.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-01-19T06:15:14.246 INFO:teuthology.orchestra.run.smithi150.stdout:(4/24): libradosstriper1-17.2.7-268.g6441305d.e 3.2 MB/s | 458 kB 00:00 2024-01-19T06:15:14.270 INFO:teuthology.orchestra.run.smithi134.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-19T06:15:14.287 INFO:teuthology.orchestra.run.smithi134.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-01-19T06:15:14.421 INFO:teuthology.orchestra.run.smithi134.stdout:(18/24): ceph-radosgw-17.2.7-268.g6441305d.el9. 5.0 MB/s | 14 MB 00:02 2024-01-19T06:15:14.721 INFO:teuthology.orchestra.run.smithi150.stdout:(5/24): librgw2-17.2.7-268.g6441305d.el9.x86_64 10 MB/s | 4.9 MB 00:00 2024-01-19T06:15:14.771 INFO:teuthology.orchestra.run.smithi150.stdout:(6/24): python3-ceph-argparse-17.2.7-268.g64413 905 kB/s | 45 kB 00:00 2024-01-19T06:15:14.822 INFO:teuthology.orchestra.run.smithi150.stdout:(7/24): python3-ceph-common-17.2.7-268.g6441305 2.3 MB/s | 116 kB 00:00 2024-01-19T06:15:14.829 INFO:teuthology.orchestra.run.smithi134.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 90 kB/s | 49 kB 00:00 2024-01-19T06:15:14.880 INFO:teuthology.orchestra.run.smithi150.stdout:(8/24): python3-cephfs-17.2.7-268.g6441305d.el9 2.6 MB/s | 156 kB 00:00 2024-01-19T06:15:14.937 INFO:teuthology.orchestra.run.smithi134.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 144 kB/s | 67 kB 00:00 2024-01-19T06:15:14.947 INFO:teuthology.orchestra.run.smithi150.stdout:(9/24): python3-rados-17.2.7-268.g6441305d.el9. 4.7 MB/s | 319 kB 00:00 2024-01-19T06:15:14.980 INFO:teuthology.orchestra.run.smithi134.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 435 kB/s | 309 kB 00:00 2024-01-19T06:15:15.014 INFO:teuthology.orchestra.run.smithi150.stdout:(10/24): python3-rbd-17.2.7-268.g6441305d.el9.x 4.3 MB/s | 293 kB 00:00 2024-01-19T06:15:15.064 INFO:teuthology.orchestra.run.smithi150.stdout:(11/24): python3-rgw-17.2.7-268.g6441305d.el9.x 1.9 MB/s | 99 kB 00:00 2024-01-19T06:15:15.206 INFO:teuthology.orchestra.run.smithi150.stdout:(12/24): ceph-radosgw-17.2.7-268.g6441305d.el9. 4.4 MB/s | 14 MB 00:03 2024-01-19T06:15:15.305 INFO:teuthology.orchestra.run.smithi134.stdout:(22/24): librbd1-17.2.7-268.g6441305d.el9.x86_6 9.0 MB/s | 2.9 MB 00:00 2024-01-19T06:15:15.323 INFO:teuthology.orchestra.run.smithi150.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 480 kB/s | 44 kB 00:00 2024-01-19T06:15:15.348 INFO:teuthology.orchestra.run.smithi150.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 561 kB/s | 159 kB 00:00 2024-01-19T06:15:15.355 INFO:teuthology.orchestra.run.smithi134.stdout:(23/24): librados2-17.2.7-268.g6441305d.el9.x86 7.4 MB/s | 3.1 MB 00:00 2024-01-19T06:15:15.388 INFO:teuthology.orchestra.run.smithi134.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-01-19T06:15:15.390 INFO:teuthology.orchestra.run.smithi150.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-01-19T06:15:15.395 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:15.395 INFO:teuthology.orchestra.run.smithi134.stdout:Total 14 MB/s | 58 MB 00:04 2024-01-19T06:15:15.396 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:15:15.416 INFO:teuthology.orchestra.run.smithi150.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-01-19T06:15:15.426 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:15:15.426 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:15:15.441 INFO:teuthology.orchestra.run.smithi150.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-01-19T06:15:15.904 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:15:15.904 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:15:16.566 INFO:teuthology.orchestra.run.smithi150.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 43 kB/s | 49 kB 00:01 2024-01-19T06:15:17.151 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:15:17.317 INFO:teuthology.orchestra.run.smithi150.stdout:(19/24): ceph-common-17.2.7-268.g6441305d.el9.x 4.3 MB/s | 23 MB 00:05 2024-01-19T06:15:17.483 INFO:teuthology.orchestra.run.smithi150.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 149 kB/s | 309 kB 00:02 2024-01-19T06:15:17.526 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-19T06:15:17.557 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:17.584 INFO:teuthology.orchestra.run.smithi150.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 66 kB/s | 67 kB 00:01 2024-01-19T06:15:17.652 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:17.736 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T06:15:17.775 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:17.863 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:17.916 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-19T06:15:17.934 INFO:teuthology.orchestra.run.smithi150.stdout:(22/24): librados2-17.2.7-268.g6441305d.el9.x86 6.8 MB/s | 3.1 MB 00:00 2024-01-19T06:15:17.946 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-19T06:15:18.167 INFO:teuthology.orchestra.run.smithi150.stdout:(23/24): librbd1-17.2.7-268.g6441305d.el9.x86_6 5.0 MB/s | 2.9 MB 00:00 2024-01-19T06:15:18.276 INFO:teuthology.orchestra.run.smithi150.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-01-19T06:15:18.282 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:18.282 INFO:teuthology.orchestra.run.smithi150.stdout:Total 9.1 MB/s | 58 MB 00:06 2024-01-19T06:15:18.283 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:15:18.316 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:15:18.316 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:15:18.382 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-19T06:15:18.420 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T06:15:18.473 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T06:15:18.511 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-19T06:15:18.554 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 10/26 2024-01-19T06:15:18.600 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T06:15:18.669 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T06:15:18.697 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:18.746 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:18.786 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-19T06:15:18.795 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:15:18.795 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:15:18.822 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-19T06:15:18.854 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-19T06:15:19.150 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-19T06:15:19.178 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T06:15:19.268 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T06:15:19.341 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 19/26 2024-01-19T06:15:19.372 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 20/26 2024-01-19T06:15:20.053 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:15:20.430 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-19T06:15:20.553 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:20.664 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:20.741 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T06:15:20.763 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:20.843 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:20.895 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-19T06:15:20.927 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-19T06:15:21.369 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-19T06:15:21.399 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T06:15:21.462 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T06:15:21.490 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-19T06:15:21.525 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 10/26 2024-01-19T06:15:21.565 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T06:15:21.633 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T06:15:21.660 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:21.709 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:21.748 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-19T06:15:21.777 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-19T06:15:21.810 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-19T06:15:21.838 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:21.878 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:22.109 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-19T06:15:22.140 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T06:15:22.214 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T06:15:22.311 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 19/26 2024-01-19T06:15:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 20/26 2024-01-19T06:15:22.406 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:22.447 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T06:15:22.532 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T06:15:22.532 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T06:15:22.532 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T06:15:22.532 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:22.557 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:24.721 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:24.880 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:25.424 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T06:15:25.458 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T06:15:25.583 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T06:15:25.583 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T06:15:25.583 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T06:15:25.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:25.618 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:31.998 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-01-19T06:15:31.999 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:33.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T06:15:33.370 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:33.371 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T06:15:33.453 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T06:15:33.453 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:34.725 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:34.725 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 1/26 2024-01-19T06:15:34.725 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:34.725 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 5/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 6/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 7/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 8/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 9/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 10/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-19T06:15:34.726 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-19T06:15:34.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 25/26 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-01-19T06:15:34.975 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:35.616 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:35.616 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:35.616 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-01-19T06:15:35.616 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.616 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.617 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:35.618 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:15:35.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-01-19T06:15:36.310 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T06:15:36.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:36.393 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T06:15:36.482 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T06:15:36.482 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:36.547 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:28 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:15:36.705 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:17.2.7-268.g6441305d.el9 ceph 48 M 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T06:15:36.706 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:17.2.7-268.g6441305d.el9 ceph 162 k 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-19T06:15:36.707 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:36.708 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 49 M 2024-01-19T06:15:36.708 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 195 M 2024-01-19T06:15:36.708 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:15:36.873 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-01-19T06:15:36.907 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-01-19T06:15:36.949 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-01-19T06:15:36.974 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-17.2.7-268.g6441305d.el9.x 612 kB/s | 162 kB 00:00 2024-01-19T06:15:36.999 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 1/26 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T06:15:37.713 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 5/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 6/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 7/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 8/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 9/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 10/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-19T06:15:37.714 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T06:15:37.715 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-19T06:15:37.716 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 25/26 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.876 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T06:15:38.877 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:38.878 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:15:39.260 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2024-01-19T06:15:39.291 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-17.2.7-268.g6441305d.el9.x86_6 19 MB/s | 48 MB 00:02 2024-01-19T06:15:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:39.295 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 49 MB 00:02 2024-01-19T06:15:39.295 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:15:39.313 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:15:39.313 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:15:39.698 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:15:39.699 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:15:39.826 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:30 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:15:39.977 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:15:39.977 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:39.977 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:15:39.977 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:17.2.7-268.g6441305d.el9 ceph 48 M 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:17.2.7-268.g6441305d.el9 ceph 162 k 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-01-19T06:15:39.978 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:39.979 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 49 M 2024-01-19T06:15:39.979 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 195 M 2024-01-19T06:15:39.979 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:15:40.144 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-01-19T06:15:40.178 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-01-19T06:15:40.220 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-19T06:15:40.244 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): libcephsqlite-17.2.7-268.g6441305d.el9.x 614 kB/s | 162 kB 00:00 2024-01-19T06:15:40.270 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-01-19T06:15:40.310 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:15:40.381 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-19T06:15:40.415 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-19T06:15:40.465 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-19T06:15:40.508 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-19T06:15:40.536 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T06:15:42.303 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-17.2.7-268.g6441305d.el9.x86_6 21 MB/s | 48 MB 00:02 2024-01-19T06:15:42.306 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:42.307 INFO:teuthology.orchestra.run.smithi150.stdout:Total 21 MB/s | 49 MB 00:02 2024-01-19T06:15:42.307 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:15:42.327 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:15:42.327 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:15:42.714 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:15:42.714 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:15:43.322 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:15:43.377 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-19T06:15:43.411 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-19T06:15:43.467 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-19T06:15:43.512 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-19T06:15:43.540 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T06:15:44.963 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T06:15:45.044 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 1/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-19T06:15:47.470 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-19T06:15:47.836 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-19T06:15:47.837 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:47.837 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:15:47.962 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T06:15:48.044 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T06:15:48.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-01-19T06:15:48.690 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:40 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:15:48.849 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:17.2.7-268.g6441305d.el9 ceph 6.5 k 2024-01-19T06:15:48.851 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.1 M 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:17.2.7-268.g6441305d.el9 ceph 1.4 M 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 239 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.7 M 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:17.2.7-268.g6441305d.el9 ceph 15 M 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T06:15:48.852 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T06:15:48.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout:Install 33 Packages 2024-01-19T06:15:48.854 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:48.855 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 27 M 2024-01-19T06:15:48.855 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 99 M 2024-01-19T06:15:48.855 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:15:49.177 INFO:teuthology.orchestra.run.smithi134.stdout:(1/33): ceph-17.2.7-268.g6441305d.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-01-19T06:15:49.419 INFO:teuthology.orchestra.run.smithi134.stdout:(2/33): ceph-mds-17.2.7-268.g6441305d.el9.x86_6 5.1 MB/s | 2.1 MB 00:00 2024-01-19T06:15:49.470 INFO:teuthology.orchestra.run.smithi134.stdout:(3/33): ceph-mgr-17.2.7-268.g6441305d.el9.x86_6 3.1 MB/s | 1.4 MB 00:00 2024-01-19T06:15:49.578 INFO:teuthology.orchestra.run.smithi134.stdout:(4/33): ceph-mgr-modules-core-17.2.7-268.g64413 2.2 MB/s | 239 kB 00:00 2024-01-19T06:15:49.687 INFO:teuthology.orchestra.run.smithi134.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-01-19T06:15:49.720 INFO:teuthology.orchestra.run.smithi134.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-01-19T06:15:49.788 INFO:teuthology.orchestra.run.smithi134.stdout:(7/33): ceph-mon-17.2.7-268.g6441305d.el9.x86_6 6.1 MB/s | 3.7 MB 00:00 2024-01-19T06:15:49.812 INFO:teuthology.orchestra.run.smithi134.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 514 kB/s | 47 kB 00:00 2024-01-19T06:15:49.838 INFO:teuthology.orchestra.run.smithi134.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-01-19T06:15:49.863 INFO:teuthology.orchestra.run.smithi134.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-19T06:15:49.922 INFO:teuthology.orchestra.run.smithi134.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-01-19T06:15:49.947 INFO:teuthology.orchestra.run.smithi134.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-01-19T06:15:49.972 INFO:teuthology.orchestra.run.smithi134.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-19T06:15:49.998 INFO:teuthology.orchestra.run.smithi134.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T06:15:50.023 INFO:teuthology.orchestra.run.smithi134.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-19T06:15:50.223 INFO:teuthology.orchestra.run.smithi134.stdout:(16/33): ceph-osd-17.2.7-268.g6441305d.el9.x86_ 19 MB/s | 15 MB 00:00 2024-01-19T06:15:50.240 INFO:teuthology.orchestra.run.smithi134.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 180 kB/s | 43 kB 00:00 2024-01-19T06:15:50.274 INFO:teuthology.orchestra.run.smithi134.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 685 kB/s | 172 kB 00:00 2024-01-19T06:15:50.291 INFO:teuthology.orchestra.run.smithi134.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-01-19T06:15:50.316 INFO:teuthology.orchestra.run.smithi134.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-01-19T06:15:50.336 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T06:15:50.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 1/6 2024-01-19T06:15:50.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2/6 2024-01-19T06:15:50.336 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-19T06:15:50.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-19T06:15:50.337 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-19T06:15:50.342 INFO:teuthology.orchestra.run.smithi134.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-01-19T06:15:50.367 INFO:teuthology.orchestra.run.smithi134.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-01-19T06:15:50.392 INFO:teuthology.orchestra.run.smithi134.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-01-19T06:15:50.418 INFO:teuthology.orchestra.run.smithi134.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-01-19T06:15:50.443 INFO:teuthology.orchestra.run.smithi134.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-01-19T06:15:50.468 INFO:teuthology.orchestra.run.smithi134.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-01-19T06:15:50.493 INFO:teuthology.orchestra.run.smithi134.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-01-19T06:15:50.519 INFO:teuthology.orchestra.run.smithi134.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-01-19T06:15:50.544 INFO:teuthology.orchestra.run.smithi134.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-01-19T06:15:50.578 INFO:teuthology.orchestra.run.smithi134.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-01-19T06:15:50.603 INFO:teuthology.orchestra.run.smithi134.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-01-19T06:15:50.628 INFO:teuthology.orchestra.run.smithi134.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-01-19T06:15:50.662 INFO:teuthology.orchestra.run.smithi134.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-01-19T06:15:50.665 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:50.665 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 27 MB 00:01 2024-01-19T06:15:50.665 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:15:50.699 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:15:50.699 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:15:50.700 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-19T06:15:50.700 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-19T06:15:50.701 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:50.702 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:15:50.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2024-01-19T06:15:51.043 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:15:51.044 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:15:51.537 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:42 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:15:51.692 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:15:51.693 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-01-19T06:15:51.693 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:15:51.693 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-01-19T06:15:51.693 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:17.2.7-268.g6441305d.el9 ceph 6.5 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.1 M 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:17.2.7-268.g6441305d.el9 ceph 1.4 M 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 239 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.7 M 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:17.2.7-268.g6441305d.el9 ceph 15 M 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T06:15:51.694 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T06:15:51.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================== 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout:Install 33 Packages 2024-01-19T06:15:51.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:51.697 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 27 M 2024-01-19T06:15:51.697 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 99 M 2024-01-19T06:15:51.697 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:15:52.007 INFO:teuthology.orchestra.run.smithi150.stdout:(1/33): ceph-17.2.7-268.g6441305d.el9.x86_64.rp 42 kB/s | 6.5 kB 00:00 2024-01-19T06:15:52.257 INFO:teuthology.orchestra.run.smithi150.stdout:(2/33): ceph-mds-17.2.7-268.g6441305d.el9.x86_6 5.1 MB/s | 2.1 MB 00:00 2024-01-19T06:15:52.266 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:15:52.298 INFO:teuthology.orchestra.run.smithi150.stdout:(3/33): ceph-mgr-17.2.7-268.g6441305d.el9.x86_6 3.1 MB/s | 1.4 MB 00:00 2024-01-19T06:15:52.331 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-19T06:15:52.363 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-19T06:15:52.382 INFO:teuthology.orchestra.run.smithi150.stdout:(4/33): ceph-mgr-modules-core-17.2.7-268.g64413 2.8 MB/s | 239 kB 00:00 2024-01-19T06:15:52.396 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-19T06:15:52.488 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-19T06:15:52.491 INFO:teuthology.orchestra.run.smithi150.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-01-19T06:15:52.516 INFO:teuthology.orchestra.run.smithi150.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-01-19T06:15:52.541 INFO:teuthology.orchestra.run.smithi150.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-01-19T06:15:52.567 INFO:teuthology.orchestra.run.smithi150.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-01-19T06:15:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-19T06:15:52.616 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-19T06:15:52.617 INFO:teuthology.orchestra.run.smithi150.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-19T06:15:52.684 INFO:teuthology.orchestra.run.smithi150.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-01-19T06:15:52.688 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-19T06:15:52.741 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-01-19T06:15:52.743 INFO:teuthology.orchestra.run.smithi150.stdout:(11/33): ceph-mon-17.2.7-268.g6441305d.el9.x86_ 5.0 MB/s | 3.7 MB 00:00 2024-01-19T06:15:52.768 INFO:teuthology.orchestra.run.smithi150.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-01-19T06:15:52.780 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2024-01-19T06:15:52.793 INFO:teuthology.orchestra.run.smithi150.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-19T06:15:52.818 INFO:teuthology.orchestra.run.smithi150.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T06:15:52.828 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-01-19T06:15:52.844 INFO:teuthology.orchestra.run.smithi150.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-19T06:15:52.885 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-01-19T06:15:53.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-01-19T06:15:53.111 INFO:teuthology.orchestra.run.smithi150.stdout:(16/33): ceph-osd-17.2.7-268.g6441305d.el9.x86_ 18 MB/s | 15 MB 00:00 2024-01-19T06:15:53.126 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-01-19T06:15:53.136 INFO:teuthology.orchestra.run.smithi150.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-01-19T06:15:53.156 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-01-19T06:15:53.191 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-01-19T06:15:53.203 INFO:teuthology.orchestra.run.smithi150.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-01-19T06:15:53.221 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-19T06:15:53.262 INFO:teuthology.orchestra.run.smithi150.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 411 kB/s | 172 kB 00:00 2024-01-19T06:15:53.287 INFO:teuthology.orchestra.run.smithi150.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-01-19T06:15:53.298 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-01-19T06:15:53.331 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T06:15:53.345 INFO:teuthology.orchestra.run.smithi150.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-01-19T06:15:53.365 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-01-19T06:15:53.371 INFO:teuthology.orchestra.run.smithi150.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-01-19T06:15:53.406 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-01-19T06:15:53.429 INFO:teuthology.orchestra.run.smithi150.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-01-19T06:15:53.440 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-01-19T06:15:53.455 INFO:teuthology.orchestra.run.smithi150.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-01-19T06:15:53.480 INFO:teuthology.orchestra.run.smithi150.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 972 kB/s | 358 kB 00:00 2024-01-19T06:15:53.485 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-01-19T06:15:53.547 INFO:teuthology.orchestra.run.smithi150.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-01-19T06:15:53.564 INFO:teuthology.orchestra.run.smithi150.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 588 kB/s | 79 kB 00:00 2024-01-19T06:15:53.575 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-01-19T06:15:53.614 INFO:teuthology.orchestra.run.smithi150.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-01-19T06:15:53.616 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-01-19T06:15:53.640 INFO:teuthology.orchestra.run.smithi150.stdout:(29/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-01-19T06:15:53.656 INFO:teuthology.orchestra.run.smithi150.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-01-19T06:15:53.698 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-01-19T06:15:53.698 INFO:teuthology.orchestra.run.smithi150.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-01-19T06:15:53.732 INFO:teuthology.orchestra.run.smithi150.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-01-19T06:15:53.766 INFO:teuthology.orchestra.run.smithi150.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-01-19T06:15:53.769 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:15:53.769 INFO:teuthology.orchestra.run.smithi150.stdout:Total 13 MB/s | 27 MB 00:02 2024-01-19T06:15:53.770 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:15:53.808 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:15:53.808 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:15:53.842 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-01-19T06:15:53.967 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T06:15:54.115 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 28/33 2024-01-19T06:15:54.152 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:15:54.152 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:15:54.155 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T06:15:54.180 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:55.348 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:15:55.410 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-19T06:15:55.442 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-19T06:15:55.475 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-19T06:15:55.567 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-19T06:15:55.573 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T06:15:55.601 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:55.653 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-19T06:15:55.695 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-19T06:15:55.768 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-19T06:15:55.820 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-01-19T06:15:55.867 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2024-01-19T06:15:55.915 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-01-19T06:15:55.971 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-01-19T06:15:55.982 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T06:15:56.006 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T06:15:56.006 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:56.006 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T06:15:56.006 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T06:15:56.006 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T06:15:56.007 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:56.163 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-01-19T06:15:56.212 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-01-19T06:15:56.216 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T06:15:56.239 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T06:15:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T06:15:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T06:15:56.240 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T06:15:56.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:56.251 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-01-19T06:15:56.287 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-01-19T06:15:56.305 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T06:15:56.316 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-19T06:15:56.391 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-01-19T06:15:56.450 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T06:15:56.484 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-01-19T06:15:56.524 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-01-19T06:15:56.559 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-01-19T06:15:56.594 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-01-19T06:15:56.674 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-01-19T06:15:56.716 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-01-19T06:15:56.789 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-01-19T06:15:56.924 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-01-19T06:15:57.042 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T06:15:57.198 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 28/33 2024-01-19T06:15:57.221 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T06:15:57.246 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T06:15:57.246 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:57.246 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T06:15:57.246 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T06:15:57.247 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T06:15:57.247 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:17.2.7-268.g6441305d.el9.x86_64 1/33 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2/33 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 3/33 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 4/33 2024-01-19T06:15:57.951 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 5/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 6/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-19T06:15:57.952 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T06:15:57.953 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-19T06:15:57.954 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-19T06:15:57.954 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-19T06:15:57.954 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-19T06:15:57.954 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-19T06:15:58.639 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T06:15:58.667 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T06:15:58.962 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T06:15:58.963 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:15:58.964 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:15:59.065 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T06:15:59.090 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:59.291 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T06:15:59.314 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T06:15:59.314 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:15:59.314 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T06:15:59.314 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T06:15:59.314 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T06:15:59.315 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:15:59.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-01-19T06:15:59.380 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T06:15:59.894 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:51 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:15:59.995 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:00.045 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:00.046 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:00.046 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:00.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-01-19T06:16:00.697 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:52 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:00.847 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:00.848 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 80 k 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 80 k 2024-01-19T06:16:00.852 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 364 k 2024-01-19T06:16:00.853 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:01.055 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-17.2.7-268.g6441305d.el9.noarch.rpm 396 kB/s | 80 kB 00:00 2024-01-19T06:16:01.056 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:01.056 INFO:teuthology.orchestra.run.smithi134.stdout:Total 392 kB/s | 80 kB 00:00 2024-01-19T06:16:01.056 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:01.059 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:01.059 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:01.061 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:01.061 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T06:16:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:17.2.7-268.g6441305d.el9.x86_64 1/33 2024-01-19T06:16:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2/33 2024-01-19T06:16:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 3/33 2024-01-19T06:16:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 4/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 5/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 6/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-19T06:16:01.068 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-19T06:16:01.069 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-19T06:16:01.070 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-19T06:16:01.142 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:01.567 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:01.667 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:02.063 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:02.083 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T06:16:02.084 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:02.085 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:02.361 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:02.423 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2024-01-19T06:16:02.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T06:16:02.980 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:53 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:03.080 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:03.129 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:03.130 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:03.131 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:03.163 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:03.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2024-01-19T06:16:03.313 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-268.g6441305d.el9 ceph 137 k 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:03.314 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:03.315 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:03.315 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:03.315 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 137 k 2024-01-19T06:16:03.315 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 379 k 2024-01-19T06:16:03.315 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:03.541 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-17.2.7-268.g6441305 604 kB/s | 137 kB 00:00 2024-01-19T06:16:03.542 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:03.542 INFO:teuthology.orchestra.run.smithi134.stdout:Total 601 kB/s | 137 kB 00:00 2024-01-19T06:16:03.542 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:03.549 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:03.549 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:03.600 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:03.601 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:03.742 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:03.744 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:54 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:03.809 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:03.833 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:03.833 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:16:03.833 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T06:16:03.833 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 80 k 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 80 k 2024-01-19T06:16:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 364 k 2024-01-19T06:16:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:04.109 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-17.2.7-268.g6441305d.el9.noarch.rpm 378 kB/s | 80 kB 00:00 2024-01-19T06:16:04.110 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:04.110 INFO:teuthology.orchestra.run.smithi150.stdout:Total 375 kB/s | 80 kB 00:00 2024-01-19T06:16:04.110 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:04.115 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:04.115 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:04.117 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:04.117 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:04.191 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:04.624 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:04.714 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:04.855 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:04.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:04.855 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:04.855 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:04.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:04.856 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:05.096 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:05.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-01-19T06:16:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T06:16:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:05.413 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:05.413 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:05.414 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:05.414 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:05.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T06:16:05.671 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:57 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:05.775 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:05.823 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:05.824 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:05.824 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:05.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T06:16:06.185 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:57 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:06.336 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-268.g6441305d.el9 ceph 137 k 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:06.337 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:06.338 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 137 k 2024-01-19T06:16:06.338 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 379 k 2024-01-19T06:16:06.338 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:06.472 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:58 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:06.590 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-17.2.7-268.g6441305 544 kB/s | 137 kB 00:00 2024-01-19T06:16:06.590 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:06.590 INFO:teuthology.orchestra.run.smithi150.stdout:Total 541 kB/s | 137 kB 00:00 2024-01-19T06:16:06.590 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:06.598 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:06.598 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:06.625 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:06.626 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 1.7 M 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 23 k 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 15 k 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-01-19T06:16:06.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout:Install 7 Packages 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:06.628 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-01-19T06:16:06.629 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 8.1 M 2024-01-19T06:16:06.629 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:06.650 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:06.651 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:06.798 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:06.849 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:06.873 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:06.873 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:16:06.873 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T06:16:06.873 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:07.178 INFO:teuthology.orchestra.run.smithi134.stdout:(1/7): ceph-grafana-dashboards-17.2.7-268.g6441 124 kB/s | 23 kB 00:00 2024-01-19T06:16:07.202 INFO:teuthology.orchestra.run.smithi134.stdout:(2/7): ceph-prometheus-alerts-17.2.7-268.g64413 69 kB/s | 15 kB 00:00 2024-01-19T06:16:07.411 INFO:teuthology.orchestra.run.smithi134.stdout:(3/7): ceph-mgr-dashboard-17.2.7-268.g6441305d. 4.0 MB/s | 1.7 MB 00:00 2024-01-19T06:16:07.553 INFO:teuthology.orchestra.run.smithi134.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 24 kB/s | 9.0 kB 00:00 2024-01-19T06:16:07.703 INFO:teuthology.orchestra.run.smithi134.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 82 kB/s | 41 kB 00:00 2024-01-19T06:16:07.795 INFO:teuthology.orchestra.run.smithi134.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 80 kB/s | 31 kB 00:00 2024-01-19T06:16:07.880 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T06:16:07.880 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:07.880 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:07.881 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:07.881 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:07.881 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:08.012 INFO:teuthology.orchestra.run.smithi134.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 411 kB/s | 188 kB 00:00 2024-01-19T06:16:08.016 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:08.016 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-19T06:16:08.016 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:08.024 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:08.024 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:08.092 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:08.093 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:08.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2024-01-19T06:16:08.387 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:08.455 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-19T06:16:08.489 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-19T06:16:08.535 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-19T06:16:08.575 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-19T06:16:08.596 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 5/7 2024-01-19T06:16:08.694 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:59 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:08.796 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:08.817 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 6/7 2024-01-19T06:16:08.848 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:08.848 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:08.848 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:08.849 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T06:16:08.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 1/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 3/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-19T06:16:09.365 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-19T06:16:09.485 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:00 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:09.636 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 1.7 M 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 23 k 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 15 k 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T06:16:09.637 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout:Install 7 Packages 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2024-01-19T06:16:09.638 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 8.1 M 2024-01-19T06:16:09.639 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:09.689 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:09.690 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:09.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T06:16:09.987 INFO:teuthology.orchestra.run.smithi150.stdout:(1/7): ceph-prometheus-alerts-17.2.7-268.g64413 79 kB/s | 15 kB 00:00 2024-01-19T06:16:10.021 INFO:teuthology.orchestra.run.smithi150.stdout:(2/7): ceph-grafana-dashboards-17.2.7-268.g6441 106 kB/s | 23 kB 00:00 2024-01-19T06:16:10.179 INFO:teuthology.orchestra.run.smithi150.stdout:(3/7): ceph-mgr-dashboard-17.2.7-268.g6441305d. 4.5 MB/s | 1.7 MB 00:00 2024-01-19T06:16:10.205 INFO:teuthology.orchestra.run.smithi150.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2024-01-19T06:16:10.230 INFO:teuthology.orchestra.run.smithi150.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 196 kB/s | 41 kB 00:00 2024-01-19T06:16:10.255 INFO:teuthology.orchestra.run.smithi150.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 409 kB/s | 31 kB 00:00 2024-01-19T06:16:10.339 INFO:teuthology.orchestra.run.smithi150.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-01-19T06:16:10.342 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:10.342 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-01-19T06:16:10.342 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:10.350 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:10.350 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:10.415 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:10.416 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:10.485 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:02 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:10.635 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:10.636 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-19T06:16:10.636 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 7.4 M 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T06:16:10.637 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout:=========================================================================================== 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:10.638 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-01-19T06:16:10.639 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-01-19T06:16:10.639 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:10.711 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:10.773 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-19T06:16:10.807 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-19T06:16:10.854 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-19T06:16:10.894 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-19T06:16:10.923 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 5/7 2024-01-19T06:16:11.035 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 32 kB 00:00 2024-01-19T06:16:11.068 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 541 kB/s | 17 kB 00:00 2024-01-19T06:16:11.103 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-01-19T06:16:11.142 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 6/7 2024-01-19T06:16:11.176 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T06:16:11.186 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-19T06:16:11.361 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-01-19T06:16:11.428 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-01-19T06:16:11.453 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-01-19T06:16:11.762 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 1/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 3/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-19T06:16:11.768 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-19T06:16:11.787 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 802 kB 00:00 2024-01-19T06:16:11.812 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 194 kB 00:00 2024-01-19T06:16:12.073 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-19T06:16:12.073 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:12.073 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:12.073 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:12.074 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:12.124 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-2 6.1 MB/s | 7.4 MB 00:01 2024-01-19T06:16:12.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T06:16:12.771 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 249 kB/s | 245 kB 00:00 2024-01-19T06:16:12.774 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:12.774 INFO:teuthology.orchestra.run.smithi134.stdout:Total 17 MB/s | 37 MB 00:02 2024-01-19T06:16:12.792 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:12.819 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:12.819 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:12.851 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:03 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:13.006 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 7.4 M 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T06:16:13.007 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================== 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2024-01-19T06:16:13.008 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:13.009 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 37 M 2024-01-19T06:16:13.009 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 209 M 2024-01-19T06:16:13.009 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:13.221 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 221 kB/s | 32 kB 00:00 2024-01-19T06:16:13.254 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 521 kB/s | 17 kB 00:00 2024-01-19T06:16:13.264 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:13.264 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:13.330 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-01-19T06:16:13.355 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 346 kB/s | 34 kB 00:00 2024-01-19T06:16:13.530 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-01-19T06:16:13.600 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-01-19T06:16:13.622 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-01-19T06:16:13.923 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-01-19T06:16:13.948 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 802 kB 00:00 2024-01-19T06:16:13.968 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:13.973 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-01-19T06:16:14.044 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T06:16:14.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T06:16:14.090 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-2 7.3 MB/s | 7.4 MB 00:01 2024-01-19T06:16:14.167 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T06:16:14.203 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T06:16:14.307 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 685 kB/s | 245 kB 00:00 2024-01-19T06:16:14.309 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:14.310 INFO:teuthology.orchestra.run.smithi150.stdout:Total 28 MB/s | 37 MB 00:01 2024-01-19T06:16:14.333 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:14.357 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:14.358 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:14.380 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T06:16:14.410 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T06:16:14.492 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T06:16:14.798 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:14.799 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:15.167 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T06:16:15.280 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T06:16:15.515 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:15.588 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T06:16:15.628 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T06:16:15.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T06:16:15.756 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T06:16:15.935 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T06:16:15.964 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T06:16:16.046 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T06:16:16.514 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T06:16:16.727 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T06:16:16.835 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T06:16:17.531 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T06:16:17.559 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T06:16:18.114 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T06:16:19.148 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T06:16:19.171 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 1/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T06:16:20.118 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T06:16:20.585 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T06:16:20.586 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T06:16:20.586 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T06:16:20.586 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T06:16:20.586 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:20.586 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:20.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-01-19T06:16:21.506 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:13 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:21.660 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:21.661 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 47 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T06:16:21.663 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-01-19T06:16:21.664 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:21.665 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-01-19T06:16:21.665 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-01-19T06:16:21.665 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:21.778 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 1/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T06:16:21.779 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T06:16:22.155 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2024-01-19T06:16:22.180 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-01-19T06:16:22.192 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T06:16:22.192 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:22.192 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:22.192 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T06:16:22.193 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:22.194 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:22.231 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-01-19T06:16:22.247 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-01-19T06:16:22.273 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-01-19T06:16:22.298 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): ceph-mgr-rook-17.2.7-268.g6441305d.el9. 175 kB/s | 47 kB 00:00 2024-01-19T06:16:22.332 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-01-19T06:16:22.457 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-01-19T06:16:22.474 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2024-01-19T06:16:22.574 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-google-auth-2.26.0-1.el9.noarc 835 kB/s | 202 kB 00:00 2024-01-19T06:16:22.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2024-01-19T06:16:22.599 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-01-19T06:16:22.666 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2024-01-19T06:16:22.700 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-01-19T06:16:22.701 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:22.702 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-01-19T06:16:22.702 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:22.714 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:22.714 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:22.809 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:22.809 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:23.162 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:14 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:23.316 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:23.317 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:23.317 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:23.317 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:23.317 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:23.317 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 47 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T06:16:23.318 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2024-01-19T06:16:23.319 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:23.323 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:23.487 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 24 kB 00:00 2024-01-19T06:16:23.495 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T06:16:23.512 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-01-19T06:16:23.542 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T06:16:23.554 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-01-19T06:16:23.579 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-01-19T06:16:23.597 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T06:16:23.605 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-19T06:16:23.630 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-01-19T06:16:23.634 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T06:16:23.655 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): ceph-mgr-rook-17.2.7-268.g6441305d.el9. 167 kB/s | 47 kB 00:00 2024-01-19T06:16:23.676 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T06:16:23.770 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T06:16:23.879 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T06:16:23.927 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T06:16:24.022 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2024-01-19T06:16:24.047 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 36 kB/s | 15 kB 00:00 2024-01-19T06:16:24.281 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-01-19T06:16:24.381 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 278 kB/s | 202 kB 00:00 2024-01-19T06:16:24.557 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 326 kB/s | 90 kB 00:00 2024-01-19T06:16:24.638 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T06:16:24.683 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T06:16:24.717 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T06:16:24.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T06:16:24.786 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T06:16:24.806 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.3 MB/s | 1.0 MB 00:00 2024-01-19T06:16:24.811 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:24.811 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-19T06:16:24.812 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:24.825 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:24.826 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:24.926 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:24.926 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:25.449 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:25.532 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T06:16:25.532 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 1/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T06:16:25.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T06:16:25.606 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T06:16:25.667 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T06:16:25.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T06:16:25.759 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T06:16:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T06:16:25.902 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T06:16:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:25.929 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:26.020 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T06:16:26.068 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T06:16:26.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T06:16:26.768 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:18 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:26.797 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T06:16:26.848 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T06:16:26.882 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T06:16:26.919 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 121 k 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T06:16:26.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-19T06:16:26.921 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:26.922 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-01-19T06:16:26.922 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-01-19T06:16:26.922 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:26.928 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T06:16:26.952 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T06:16:27.242 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 850 kB/s | 228 kB 00:00 2024-01-19T06:16:27.393 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-mgr-cephadm-17.2.7-268.g6441305d.el 290 kB/s | 121 kB 00:00 2024-01-19T06:16:27.585 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 301 kB/s | 58 kB 00:00 2024-01-19T06:16:27.652 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 769 kB/s | 51 kB 00:00 2024-01-19T06:16:27.752 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 1/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T06:16:27.784 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T06:16:27.785 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T06:16:27.785 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T06:16:28.206 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T06:16:28.207 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T06:16:28.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T06:16:28.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T06:16:28.208 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T06:16:28.208 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:28.208 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:28.210 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 5.8 MB 00:01 2024-01-19T06:16:28.212 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:28.212 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-01-19T06:16:28.213 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:28.222 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:28.222 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:28.311 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:28.311 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:28.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T06:16:28.606 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:28.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T06:16:28.827 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T06:16:29.032 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:20 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:29.186 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 121 k 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T06:16:29.187 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 6.8 M 2024-01-19T06:16:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2024-01-19T06:16:29.189 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:29.345 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T06:16:29.438 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T06:16:29.502 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T06:16:29.532 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T06:16:29.542 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-19T06:16:29.675 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-01-19T06:16:29.701 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): ceph-mgr-cephadm-17.2.7-268.g6441305d.el 371 kB/s | 121 kB 00:00 2024-01-19T06:16:29.801 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 461 kB/s | 58 kB 00:00 2024-01-19T06:16:29.826 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-01-19T06:16:29.851 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-01-19T06:16:29.853 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:29.853 INFO:teuthology.orchestra.run.smithi150.stdout:Total 10 MB/s | 6.8 MB 00:00 2024-01-19T06:16:29.854 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:29.862 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:29.863 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:29.952 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:29.953 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:30.266 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:30.309 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T06:16:30.309 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/6 2024-01-19T06:16:30.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T06:16:30.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T06:16:30.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T06:16:30.310 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T06:16:30.452 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T06:16:30.503 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T06:16:30.626 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T06:16:30.626 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:30.626 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:30.626 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:30.626 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:30.627 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:30.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-01-19T06:16:31.019 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T06:16:31.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T06:16:31.168 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T06:16:31.207 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T06:16:31.446 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:23 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:31.596 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 811 k 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T06:16:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 889 k 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.6 M 2024-01-19T06:16:31.598 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:31.711 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 696 kB/s | 79 kB 00:00 2024-01-19T06:16:31.903 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-17.2.7-268.g6441305d.el9.x86_6 2.6 MB/s | 811 kB 00:00 2024-01-19T06:16:31.904 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:31.904 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.8 MB/s | 889 kB 00:00 2024-01-19T06:16:31.905 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:31.924 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:31.924 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:31.999 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:32.032 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T06:16:32.032 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/6 2024-01-19T06:16:32.032 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T06:16:32.032 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T06:16:32.033 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T06:16:32.033 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T06:16:32.180 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:32.315 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T06:16:32.335 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:32.336 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:32.378 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T06:16:32.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2024-01-19T06:16:33.084 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T06:16:33.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/2 2024-01-19T06:16:33.131 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:24 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:33.281 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 811 k 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:33.282 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2024-01-19T06:16:33.283 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:33.283 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 889 k 2024-01-19T06:16:33.283 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 2.6 M 2024-01-19T06:16:33.283 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:33.393 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:33.411 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 617 kB/s | 79 kB 00:00 2024-01-19T06:16:33.620 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-17.2.7-268.g6441305d.el9.x86_6 2.4 MB/s | 811 kB 00:00 2024-01-19T06:16:33.621 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:33.621 INFO:teuthology.orchestra.run.smithi150.stdout:Total 2.6 MB/s | 889 kB 00:00 2024-01-19T06:16:33.621 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:33.641 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:33.642 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:33.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-01-19T06:16:33.718 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:33.718 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:33.896 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:34.031 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T06:16:34.111 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T06:16:34.238 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:34.391 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 126 k 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 126 k 2024-01-19T06:16:34.392 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 448 k 2024-01-19T06:16:34.393 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:34.615 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-17.2.7-268.g6441305d.el9.x86_64. 568 kB/s | 126 kB 00:00 2024-01-19T06:16:34.615 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:34.616 INFO:teuthology.orchestra.run.smithi134.stdout:Total 565 kB/s | 126 kB 00:00 2024-01-19T06:16:34.616 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:34.621 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:34.621 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:34.677 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:34.677 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:34.833 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:34.838 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T06:16:34.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/2 2024-01-19T06:16:34.929 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:35.160 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T06:16:35.160 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:35.160 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:35.160 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T06:16:35.160 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:35.161 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:35.386 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:35.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:35.679 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:35.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-01-19T06:16:36.017 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:27 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:36.167 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:36.168 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:36.168 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-01-19T06:16:36.168 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 126 k 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 126 k 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 448 k 2024-01-19T06:16:36.169 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:36.374 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-17.2.7-268.g6441305d.el9.x86_64. 618 kB/s | 126 kB 00:00 2024-01-19T06:16:36.374 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:36.374 INFO:teuthology.orchestra.run.smithi150.stdout:Total 614 kB/s | 126 kB 00:00 2024-01-19T06:16:36.374 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:36.380 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:36.380 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:36.434 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:36.434 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:36.515 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:28 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:36.594 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:36.618 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:36.667 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:36.668 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:36.669 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:36.688 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:36.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-01-19T06:16:37.136 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:37.309 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:37.461 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:37.461 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:37.461 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:37.462 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 25 k 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 110 k 2024-01-19T06:16:37.463 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:37.651 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-17.2.7-268.g6441305d.el9.x86_64 133 kB/s | 25 kB 00:00 2024-01-19T06:16:37.652 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:37.652 INFO:teuthology.orchestra.run.smithi134.stdout:Total 131 kB/s | 25 kB 00:00 2024-01-19T06:16:37.652 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:37.655 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:37.655 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:37.657 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:37.657 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:37.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2024-01-19T06:16:37.746 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:37.848 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:38.224 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:38.274 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:29 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:38.378 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:38.428 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:38.429 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:38.429 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:38.491 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:38.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2024-01-19T06:16:38.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-01-19T06:16:39.068 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:30 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:39.220 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:39.220 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:39.220 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 25 k 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 110 k 2024-01-19T06:16:39.221 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:39.274 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:39.373 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:39.377 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-17.2.7-268.g6441305d.el9.x86_64 160 kB/s | 25 kB 00:00 2024-01-19T06:16:39.377 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:39.377 INFO:teuthology.orchestra.run.smithi150.stdout:Total 159 kB/s | 25 kB 00:00 2024-01-19T06:16:39.378 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:39.381 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:39.381 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:39.382 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:39.383 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:39.422 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:39.423 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:39.424 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:39.463 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:39.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-01-19T06:16:39.582 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:39.983 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:40.063 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:40.162 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:40.211 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:40.212 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:40.213 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:40.259 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:40.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-01-19T06:16:40.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2024-01-19T06:16:40.848 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:40.948 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:40.998 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:40.998 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:40.999 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:41.013 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:41.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-01-19T06:16:41.113 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:41.163 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:41.164 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:41.164 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:41.237 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2024-01-19T06:16:41.648 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:33 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:41.749 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:41.799 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:41.800 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:41.801 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:41.801 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:41.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-01-19T06:16:41.900 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:41.950 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:41.951 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:41.952 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:42.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2024-01-19T06:16:42.454 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:34 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:42.555 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:42.590 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:33 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:42.606 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:42.607 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:42.607 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:42.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-01-19T06:16:42.692 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:42.742 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:42.743 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:42.743 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:42.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2024-01-19T06:16:43.247 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:35 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:43.349 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:43.370 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:34 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:43.398 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:43.399 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-19T06:16:43.399 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:43.470 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:43.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-01-19T06:16:43.520 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:43.521 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:43.521 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:43.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2024-01-19T06:16:44.049 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:44.148 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:35 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:44.199 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:44.199 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 81 k 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:44.200 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:44.201 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 81 k 2024-01-19T06:16:44.201 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 222 k 2024-01-19T06:16:44.201 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:44.248 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:44.298 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:44.299 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:44.299 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:44.382 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2024-01-19T06:16:44.404 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-17.2.7-268.g6441305d.el9.x86_64.rpm 401 kB/s | 81 kB 00:00 2024-01-19T06:16:44.404 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:44.405 INFO:teuthology.orchestra.run.smithi134.stdout:Total 399 kB/s | 81 kB 00:00 2024-01-19T06:16:44.405 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:44.411 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:44.411 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:44.466 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:44.466 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:44.611 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:44.946 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:35 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:45.046 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T06:16:45.096 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:45.097 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-01-19T06:16:45.098 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:45.100 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:45.174 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2024-01-19T06:16:45.399 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:45.399 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:45.400 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:45.400 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:45.400 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:45.400 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:45.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-01-19T06:16:45.737 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:36 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:45.887 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 81 k 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:45.888 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:45.889 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 81 k 2024-01-19T06:16:45.889 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 222 k 2024-01-19T06:16:45.889 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:46.130 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-17.2.7-268.g6441305d.el9.x86_64.rpm 337 kB/s | 81 kB 00:00 2024-01-19T06:16:46.131 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:46.131 INFO:teuthology.orchestra.run.smithi150.stdout:Total 336 kB/s | 81 kB 00:00 2024-01-19T06:16:46.131 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:46.138 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:46.138 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:46.192 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:46.193 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:46.225 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:38 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:46.336 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:46.381 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:46.382 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.9 M 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 12 M 2024-01-19T06:16:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:46.445 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:46.790 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-17.2.7-268.g6441305d.el9.x86_64.rpm 7.2 MB/s | 2.9 MB 00:00 2024-01-19T06:16:46.790 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:46.791 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.2 MB/s | 2.9 MB 00:00 2024-01-19T06:16:46.791 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:46.800 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:46.800 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:46.855 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:46.870 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:46.871 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:47.175 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:47.349 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T06:16:47.373 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:47.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2024-01-19T06:16:48.001 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:39 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:48.156 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:48.157 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:48.158 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:48.158 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:48.158 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.9 M 2024-01-19T06:16:48.158 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 12 M 2024-01-19T06:16:48.158 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:48.518 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:48.518 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:48.519 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:48.519 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:48.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:48.519 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:48.541 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-17.2.7-268.g6441305d.el9.x86_64.rpm 7.7 MB/s | 2.9 MB 00:00 2024-01-19T06:16:48.542 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:48.542 INFO:teuthology.orchestra.run.smithi150.stdout:Total 7.6 MB/s | 2.9 MB 00:00 2024-01-19T06:16:48.543 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:48.551 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:48.552 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:48.624 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:48.625 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:48.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-01-19T06:16:49.030 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:49.101 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:49.125 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:49.125 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T06:16:49.125 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T06:16:49.125 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T06:16:49.126 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T06:16:49.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:49.379 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Fri 19 Jan 2024 06:15:08 AM UTC. 2024-01-19T06:16:49.530 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 164 k 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-19T06:16:49.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:49.532 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 164 k 2024-01-19T06:16:49.532 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 477 k 2024-01-19T06:16:49.532 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-19T06:16:49.800 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-17.2.7-268.g6441305d.el9.x86_64.rpm 612 kB/s | 164 kB 00:00 2024-01-19T06:16:49.801 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:49.801 INFO:teuthology.orchestra.run.smithi134.stdout:Total 609 kB/s | 164 kB 00:00 2024-01-19T06:16:49.801 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-19T06:16:49.808 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-19T06:16:49.808 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-19T06:16:49.863 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-19T06:16:49.863 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-19T06:16:50.018 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-19T06:16:50.114 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:50.234 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:50.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:50.234 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:50.234 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:50.235 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:50.235 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:50.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2024-01-19T06:16:50.551 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:16:50.893 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-19T06:16:51.075 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:42 ago on Fri 19 Jan 2024 06:15:09 AM UTC. 2024-01-19T06:16:51.141 DEBUG:teuthology.parallel:result is None 2024-01-19T06:16:51.228 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 164 k 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-01-19T06:16:51.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:51.230 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 164 k 2024-01-19T06:16:51.230 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 477 k 2024-01-19T06:16:51.230 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-01-19T06:16:51.511 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-17.2.7-268.g6441305d.el9.x86_64.rpm 583 kB/s | 164 kB 00:00 2024-01-19T06:16:51.512 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:16:51.512 INFO:teuthology.orchestra.run.smithi150.stdout:Total 580 kB/s | 164 kB 00:00 2024-01-19T06:16:51.512 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-01-19T06:16:51.520 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-01-19T06:16:51.520 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-01-19T06:16:51.575 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-01-19T06:16:51.575 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-01-19T06:16:51.720 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-01-19T06:16:51.826 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:52.290 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:52.584 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T06:16:52.585 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:52.585 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-01-19T06:16:52.585 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T06:16:52.585 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:16:52.585 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-01-19T06:16:52.839 DEBUG:teuthology.parallel:result is None 2024-01-19T06:16:52.840 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:16:52.840 INFO:teuthology.packaging:ref: None 2024-01-19T06:16:52.840 INFO:teuthology.packaging:tag: None 2024-01-19T06:16:52.840 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:16:52.840 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:16:52.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T06:16:53.009 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T06:16:53.035 INFO:teuthology.orchestra.run.smithi134.stdout:17.2.7-268.g6441305d.el9 2024-01-19T06:16:53.036 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-268.g6441305d.el9 2024-01-19T06:16:53.037 INFO:teuthology.task.install:The correct ceph version 17.2.7-268.g6441305d is installed. 2024-01-19T06:16:53.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T06:16:53.040 INFO:teuthology.packaging:ref: None 2024-01-19T06:16:53.040 INFO:teuthology.packaging:tag: None 2024-01-19T06:16:53.040 INFO:teuthology.packaging:branch: quincy 2024-01-19T06:16:53.041 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:16:53.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T06:16:53.229 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T06:16:53.252 INFO:teuthology.orchestra.run.smithi150.stdout:17.2.7-268.g6441305d.el9 2024-01-19T06:16:53.252 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-268.g6441305d.el9 2024-01-19T06:16:53.252 INFO:teuthology.task.install:The correct ceph version 17.2.7-268.g6441305d is installed. 2024-01-19T06:16:53.254 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-19T06:16:53.254 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:16:53.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T06:16:53.286 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:16:53.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T06:16:53.322 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-19T06:16:53.322 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:16:53.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T06:16:53.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T06:16:53.426 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:16:53.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T06:16:53.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T06:16:53.524 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-19T06:16:53.524 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:16:53.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T06:16:53.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T06:16:53.621 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:16:53.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T06:16:53.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T06:16:53.716 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-19T06:16:53.716 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:16:53.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T06:16:53.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T06:16:53.811 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:16:53.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T06:16:53.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T06:16:53.907 INFO:teuthology.run_tasks:Running task print... 2024-01-19T06:16:53.916 INFO:teuthology.task.print:**** done install task... 2024-01-19T06:16:53.917 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-19T06:16:54.014 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-19T06:16:54.014 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-01-19T06:16:54.015 INFO:tasks.cephadm:Cluster fsid is 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:16:54.015 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-19T06:16:54.015 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-19T06:16:54.015 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi134': '172.21.15.134', 'mon.smithi150': '172.21.15.150'} 2024-01-19T06:16:54.015 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-19T06:16:54.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-01-19T06:16:54.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-01-19T06:16:54.070 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-01-19T06:16:54.071 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.345 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 19 06:16 /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.345 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.518 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 19 06:16 /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.518 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.534 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-19T06:16:54.567 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-01-19T06:16:54.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-19T06:16:54.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-19T06:16:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T06:16:55.522 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd", 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d" 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-01-19T06:17:53.332 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-01-19T06:17:53.376 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-01-19T06:17:53.376 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-19T06:17:53.376 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd", 2024-01-19T06:17:53.376 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-01-19T06:17:53.376 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d" 2024-01-19T06:17:53.377 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-01-19T06:17:53.377 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-01-19T06:17:53.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-01-19T06:17:53.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-01-19T06:17:53.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-01-19T06:17:53.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-01-19T06:17:53.528 INFO:tasks.cephadm:Writing seed config... 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-19T06:17:53.530 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-19T06:17:53.531 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-19T06:17:53.532 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-19T06:17:53.533 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-19T06:17:53.533 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:17:53.533 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-19T06:17:53.548 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 573cd99a-b692-11ee-95ae-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-19T06:17:53.548 DEBUG:teuthology.orchestra.run.smithi134:mon.smithi134> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service 2024-01-19T06:17:53.590 INFO:tasks.cephadm:Bootstrapping... 2024-01-19T06:17:53.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 573cd99a-b692-11ee-95ae-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.134 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-19T06:17:53.792 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-19T06:17:53.792 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '573cd99a-b692-11ee-95ae-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.134', '--skip-admin-label'] 2024-01-19T06:17:53.818 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.8.1 2024-01-19T06:17:53.818 INFO:teuthology.orchestra.run.smithi134.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-01-19T06:17:53.819 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying podman|docker is present... 2024-01-19T06:17:53.842 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.8.1 2024-01-19T06:17:53.842 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying lvm2 is present... 2024-01-19T06:17:53.842 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying time synchronization is in place... 2024-01-19T06:17:53.850 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-19T06:17:53.850 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-19T06:17:53.856 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-19T06:17:53.856 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:17:53.863 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-01-19T06:17:53.869 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-01-19T06:17:53.869 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-01-19T06:17:53.869 INFO:teuthology.orchestra.run.smithi134.stdout:Repeating the final host check... 2024-01-19T06:17:53.892 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.8.1 2024-01-19T06:17:53.892 INFO:teuthology.orchestra.run.smithi134.stdout:podman (/bin/podman) version 4.8.1 is present 2024-01-19T06:17:53.892 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl is present 2024-01-19T06:17:53.892 INFO:teuthology.orchestra.run.smithi134.stdout:lvcreate is present 2024-01-19T06:17:53.898 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-19T06:17:53.898 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-19T06:17:53.904 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-19T06:17:53.904 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:17:53.911 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Host looks OK 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Cluster fsid: 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Acquiring lock 140024138018880 on /run/cephadm/573cd99a-b692-11ee-95ae-87774f69a715.lock 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Lock 140024138018880 acquired on /run/cephadm/573cd99a-b692-11ee-95ae-87774f69a715.lock 2024-01-19T06:17:53.918 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 3300 ... 2024-01-19T06:17:53.919 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 6789 ... 2024-01-19T06:17:53.919 INFO:teuthology.orchestra.run.smithi134.stdout:Base mon IP(s) is [172.21.15.134:3300, 172.21.15.134:6789], mon addrv is [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-01-19T06:17:53.924 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.134 metric 100 2024-01-19T06:17:53.924 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.134 metric 100 2024-01-19T06:17:53.925 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-01-19T06:17:53.925 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-19T06:17:53.925 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-01-19T06:17:53.927 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-19T06:17:53.927 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-01-19T06:17:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-19T06:17:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-19T06:17:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-01-19T06:17:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-19T06:17:53.928 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-01-19T06:17:53.929 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-01-19T06:17:53.929 INFO:teuthology.orchestra.run.smithi134.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-19T06:17:53.929 INFO:teuthology.orchestra.run.smithi134.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-19T06:17:53.929 INFO:teuthology.orchestra.run.smithi134.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Getting image source signatures 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:aebaf716154c1970d30484d8c64e6de04feea816ffee6c602dc9eb55fbe60ece 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-19T06:17:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying config sha256:c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd 2024-01-19T06:17:54.432 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Writing manifest to image destination 2024-01-19T06:17:55.510 INFO:teuthology.orchestra.run.smithi134.stdout:ceph: stdout ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable) 2024-01-19T06:17:55.510 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph version: ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable) 2024-01-19T06:17:55.510 INFO:teuthology.orchestra.run.smithi134.stdout:Extracting ceph user uid/gid from container image... 2024-01-19T06:17:56.507 INFO:teuthology.orchestra.run.smithi134.stdout:stat: stdout 167 167 2024-01-19T06:17:56.508 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial keys... 2024-01-19T06:17:57.396 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQAUFKpl9CUiMBAAGH/4w/lUUVrWk8ZN/2PSKA== 2024-01-19T06:17:58.234 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQAVFKpls812LBAAKx3wT3ev5p5ejEwpHiRv+w== 2024-01-19T06:17:59.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQAWFKplKHHDIRAAR/wzXu7CSZXHDIapEfTj2A== 2024-01-19T06:17:59.190 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial monmap... 2024-01-19T06:17:59.985 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool for smithi134 [v2:172.21.15.134:3300,v1:172.21.15.134:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = octopus 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: set fsid to 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:17:59.986 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mon... 2024-01-19T06:18:00.874 INFO:teuthology.orchestra.run.smithi134.stdout:create mon.smithi134 on 2024-01-19T06:18:01.083 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-01-19T06:18:01.246 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-19T06:18:01.400 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target. 2024-01-19T06:18:01.401 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target. 2024-01-19T06:18:01.588 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134 2024-01-19T06:18:01.588 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service: Unit ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service not loaded. 2024-01-19T06:18:01.762 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target.wants/ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service → /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715@.service. 2024-01-19T06:18:01.820 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:01 smithi134 systemd[1]: Starting Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T06:18:02.079 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:01 smithi134 podman[27893]: 2024-01-19 06:18:01.982282985 +0000 UTC m=+0.086279046 container create 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240102) 2024-01-19T06:18:02.079 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 podman[27893]: 2024-01-19 06:18:01.913515867 +0000 UTC m=+0.017511923 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph:quincy 2024-01-19T06:18:02.288 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T06:18:02.288 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-01-19T06:18:02.295 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T06:18:02.295 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:18:02.296 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-01-19T06:18:02.296 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-01-19T06:18:02.296 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon to start... 2024-01-19T06:18:02.296 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon... 2024-01-19T06:18:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 podman[27893]: 2024-01-19 06:18:02.176060415 +0000 UTC m=+0.280056467 container init 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240102, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-01-19T06:18:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 podman[27893]: 2024-01-19 06:18:02.17896663 +0000 UTC m=+0.282962682 container start 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, ceph=True, RELEASE=quincy-6441305, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.build-date=20240102, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-19T06:18:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 ceph-mon[27912]: mkfs 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:02.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 ceph-mon[27912]: mon.smithi134 is new leader, mons smithi134 in quorum (ranks 0) 2024-01-19T06:18:02.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 bash[27893]: 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 2024-01-19T06:18:02.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:02 smithi134 systemd[1]: Started Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: mon.smithi134 is new leader, mons smithi134 in quorum (ranks 0) 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: monmap e1: 1 mons at {smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: fsmap 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: osdmap e1: 0 total, 0 up, 0 in 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: mgrmap e1: no daemons active 2024-01-19T06:18:03.331 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:03 smithi134 ceph-mon[27912]: from='client.? 172.21.15.134:0/1801606298' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-19T06:18:03.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout cluster: 2024-01-19T06:18:03.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout id: 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:03.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout services: 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi134 (age 0.56145s) 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout data: 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pgs: 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.721+0000 7f88ac12e700 1 Processor -- start 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88ac12e700 1 -- start start 2024-01-19T06:18:03.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88ac12e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88ac12e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4101b20 con 0x7f88a41011d0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88a9eca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88a9eca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56710/0 (socket says 172.21.15.134:56710) 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.722+0000 7f88a9eca700 1 -- 172.21.15.134:0/2507843646 learned_addr learned my addr 172.21.15.134:0/2507843646 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88a9eca700 1 -- 172.21.15.134:0/2507843646 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4101c60 con 0x7f88a41011d0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88a9eca700 1 --2- 172.21.15.134:0/2507843646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f889c00d0d0 tx=0x7f889c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da4de408f64f8db server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88a8ec8700 1 -- 172.21.15.134:0/2507843646 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c00ddf0 con 0x7f88a41011d0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88a8ec8700 1 -- 172.21.15.134:0/2507843646 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c00b0d0 con 0x7f88a41011d0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88a8ec8700 1 -- 172.21.15.134:0/2507843646 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c00b3e0 con 0x7f88a41011d0 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88ac12e700 1 -- 172.21.15.134:0/2507843646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 msgr2=0x7f88a41015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.723+0000 7f88ac12e700 1 --2- 172.21.15.134:0/2507843646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f889c00d0d0 tx=0x7f889c00ba50 comp rx=0 tx=0).stop 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- 172.21.15.134:0/2507843646 shutdown_connections 2024-01-19T06:18:03.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 --2- 172.21.15.134:0/2507843646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a41015e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- 172.21.15.134:0/2507843646 >> 172.21.15.134:0/2507843646 conn(0x7f88a40fc950 msgr2=0x7f88a40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- 172.21.15.134:0/2507843646 shutdown_connections 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- 172.21.15.134:0/2507843646 wait complete. 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 Processor -- start 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- start start 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88ac12e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a410ecd0 con 0x7f88a41011d0 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88a9eca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88a9eca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56722/0 (socket says 172.21.15.134:56722) 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.724+0000 7f88a9eca700 1 -- 172.21.15.134:0/1801606298 learned_addr learned my addr 172.21.15.134:0/1801606298 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f88a9eca700 1 -- 172.21.15.134:0/1801606298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c00bcf0 con 0x7f88a41011d0 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f88a9eca700 1 --2- 172.21.15.134:0/1801606298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f889c000f80 tx=0x7f889c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c009db0 con 0x7f88a41011d0 2024-01-19T06:18:03.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c014430 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a410eed0 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c014720 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a410fb70 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f889c01b030 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.725+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f889c028400 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.726+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a405f270 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.728+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f889c0476d0 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.797+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f88a4102540 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.797+0000 7f889a7fc700 1 -- 172.21.15.134:0/1801606298 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f889c036070 con 0x7f88a41011d0 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 msgr2=0x7f88a410e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 --2- 172.21.15.134:0/1801606298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f889c000f80 tx=0x7f889c019000 comp rx=0 tx=0).stop 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 shutdown_connections 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 --2- 172.21.15.134:0/1801606298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88a41011d0 0x7f88a410e790 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 >> 172.21.15.134:0/1801606298 conn(0x7f88a40fc950 msgr2=0x7f88a40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:03.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 shutdown_connections 2024-01-19T06:18:03.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:02.798+0000 7f88ac12e700 1 -- 172.21.15.134:0/1801606298 wait complete. 2024-01-19T06:18:03.346 INFO:teuthology.orchestra.run.smithi134.stdout:mon is available 2024-01-19T06:18:03.346 INFO:teuthology.orchestra.run.smithi134.stdout:Assimilating anything we can from ceph.conf... 2024-01-19T06:18:04.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:04.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-01-19T06:18:04.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:04.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-01-19T06:18:04.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb56511700 1 Processor -- start 2024-01-19T06:18:04.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb56511700 1 -- start start 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb56511700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb56511700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50104310 con 0x7feb501039c0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56734/0 (socket says 172.21.15.134:56734) 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.794+0000 7feb4ffff700 1 -- 172.21.15.134:0/44636640 learned_addr learned my addr 172.21.15.134:0/44636640 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb4ffff700 1 -- 172.21.15.134:0/44636640 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb50101fb0 con 0x7feb501039c0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb4ffff700 1 --2- 172.21.15.134:0/44636640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feb400098e0 tx=0x7feb40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419889e984aba3e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb4effd700 1 -- 172.21.15.134:0/44636640 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4000b860 con 0x7feb501039c0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb4effd700 1 -- 172.21.15.134:0/44636640 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4000b9c0 con 0x7feb501039c0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb4effd700 1 -- 172.21.15.134:0/44636640 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4000bc90 con 0x7feb501039c0 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb56511700 1 -- 172.21.15.134:0/44636640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 msgr2=0x7feb50103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.795+0000 7feb56511700 1 --2- 172.21.15.134:0/44636640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7feb400098e0 tx=0x7feb40009250 comp rx=0 tx=0).stop 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- 172.21.15.134:0/44636640 shutdown_connections 2024-01-19T06:18:04.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 --2- 172.21.15.134:0/44636640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50103dd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- 172.21.15.134:0/44636640 >> 172.21.15.134:0/44636640 conn(0x7feb500fda40 msgr2=0x7feb500ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- 172.21.15.134:0/44636640 shutdown_connections 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- 172.21.15.134:0/44636640 wait complete. 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 Processor -- start 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- start start 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb56511700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50077190 con 0x7feb501039c0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.796+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56742/0 (socket says 172.21.15.134:56742) 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4ffff700 1 -- 172.21.15.134:0/2036793349 learned_addr learned my addr 172.21.15.134:0/2036793349 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4ffff700 1 -- 172.21.15.134:0/2036793349 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb400094f0 con 0x7feb501039c0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4ffff700 1 --2- 172.21.15.134:0/2036793349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feb40003080 tx=0x7feb40016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4001c5c0 con 0x7feb501039c0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4001c720 con 0x7feb501039c0 2024-01-19T06:18:04.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4001ca30 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb50077390 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb500777a0 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7feb4001a450 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.797+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7feb4002e790 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.798+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5005f270 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.800+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7feb4001a890 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.868+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7feb501022c0 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.872+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7feb4002ee60 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.872+0000 7feb4cff9700 1 -- 172.21.15.134:0/2036793349 <== mon.0 v2:172.21.15.134:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7feb501022c0 con 0x7feb501039c0 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 msgr2=0x7feb50078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 --2- 172.21.15.134:0/2036793349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7feb40003080 tx=0x7feb40016000 comp rx=0 tx=0).stop 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 shutdown_connections 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 --2- 172.21.15.134:0/2036793349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb501039c0 0x7feb50078d10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:04.374 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 >> 172.21.15.134:0/2036793349 conn(0x7feb500fda40 msgr2=0x7feb50105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:04.375 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 shutdown_connections 2024-01-19T06:18:04.375 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:03.873+0000 7feb56511700 1 -- 172.21.15.134:0/2036793349 wait complete. 2024-01-19T06:18:04.375 INFO:teuthology.orchestra.run.smithi134.stdout:Generating new minimal ceph.conf... 2024-01-19T06:18:04.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:04 smithi134 ceph-mon[27912]: from='client.? 172.21.15.134:0/2036793349' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-19T06:18:04.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:04 smithi134 ceph-mon[27912]: from='client.? 172.21.15.134:0/2036793349' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-19T06:18:05.375 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:05 smithi134 ceph-mon[27912]: from='client.? 172.21.15.134:0/3359603889' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.740+0000 7f1bacd49700 1 Processor -- start 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.740+0000 7f1bacd49700 1 -- start start 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1bacd49700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1bacd49700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8104310 con 0x7f1ba81039c0 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1ba659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1ba659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56756/0 (socket says 172.21.15.134:56756) 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1ba659c700 1 -- 172.21.15.134:0/4105630914 learned_addr learned my addr 172.21.15.134:0/4105630914 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1ba659c700 1 -- 172.21.15.134:0/4105630914 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8101fb0 con 0x7f1ba81039c0 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.741+0000 7f1ba659c700 1 --2- 172.21.15.134:0/4105630914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0098e0 tx=0x7f1b9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53af85a2822024e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:05.385 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1ba559a700 1 -- 172.21.15.134:0/4105630914 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9c00b860 con 0x7f1ba81039c0 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1ba559a700 1 -- 172.21.15.134:0/4105630914 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1b9c00be30 con 0x7f1ba81039c0 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1ba559a700 1 -- 172.21.15.134:0/4105630914 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9c010380 con 0x7f1ba81039c0 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 -- 172.21.15.134:0/4105630914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 msgr2=0x7f1ba8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 --2- 172.21.15.134:0/4105630914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0098e0 tx=0x7f1b9c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 -- 172.21.15.134:0/4105630914 shutdown_connections 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 --2- 172.21.15.134:0/4105630914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba8103dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 -- 172.21.15.134:0/4105630914 >> 172.21.15.134:0/4105630914 conn(0x7f1ba80fda40 msgr2=0x7f1ba80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 -- 172.21.15.134:0/4105630914 shutdown_connections 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.742+0000 7f1bacd49700 1 -- 172.21.15.134:0/4105630914 wait complete. 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1bacd49700 1 Processor -- start 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1bacd49700 1 -- start start 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1bacd49700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1bacd49700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8111630 con 0x7f1ba81039c0 2024-01-19T06:18:05.386 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1ba659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1ba659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56766/0 (socket says 172.21.15.134:56766) 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1ba659c700 1 -- 172.21.15.134:0/3359603889 learned_addr learned my addr 172.21.15.134:0/3359603889 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1ba659c700 1 -- 172.21.15.134:0/3359603889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0094f0 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.743+0000 7f1ba659c700 1 --2- 172.21.15.134:0/3359603889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c000c00 tx=0x7f1b9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9c010a80 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1b9c01a430 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba8111830 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba8111c40 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b9c0245b0 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1b9c024af0 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.744+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1b9c02eb50 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.745+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba8102d60 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.747+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1b9c029070 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.813+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1ba810adb0 con 0x7f1ba81039c0 2024-01-19T06:18:05.387 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.813+0000 7f1b96ffd700 1 -- 172.21.15.134:0/3359603889 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f1b9c010be0 con 0x7f1ba81039c0 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 msgr2=0x7f1ba81110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 --2- 172.21.15.134:0/3359603889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c000c00 tx=0x7f1b9c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 shutdown_connections 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 --2- 172.21.15.134:0/3359603889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ba81039c0 0x7f1ba81110f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 >> 172.21.15.134:0/3359603889 conn(0x7f1ba80fda40 msgr2=0x7f1ba80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 shutdown_connections 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:04.814+0000 7f1bacd49700 1 -- 172.21.15.134:0/3359603889 wait complete. 2024-01-19T06:18:05.388 INFO:teuthology.orchestra.run.smithi134.stdout:Restarting the monitor... 2024-01-19T06:18:05.646 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:05 smithi134 systemd[1]: Stopping Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T06:18:05.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[27908]: 2024-01-19T06:18:05.645+0000 7f393a097700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi134 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T06:18:05.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[27908]: 2024-01-19T06:18:05.645+0000 7f393a097700 -1 mon.smithi134@0(leader) e1 *** Got Signal Terminated *** 2024-01-19T06:18:05.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:05 smithi134 podman[28155]: 2024-01-19 06:18:05.800010639 +0000 UTC m=+0.260632329 container died 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.build-date=20240102, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-6441305, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-01-19T06:18:06.395 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28155]: 2024-01-19 06:18:06.154247292 +0000 UTC m=+0.614868982 container cleanup 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240102, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3) 2024-01-19T06:18:06.395 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 bash[28155]: ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134 2024-01-19T06:18:06.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28169]: 2024-01-19 06:18:06.396671352 +0000 UTC m=+0.595963549 container remove 45819eb218febb0e6f624aa45bfe01467b69320a0f1d165651212540a40a7439 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240102, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-19T06:18:06.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service: Deactivated successfully. 2024-01-19T06:18:06.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 systemd[1]: Stopped Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T06:18:06.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 systemd[1]: Starting Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T06:18:06.943 INFO:teuthology.orchestra.run.smithi134.stdout:Setting mon public_network to 172.21.0.0/20 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28264]: 2024-01-19 06:18:06.714127239 +0000 UTC m=+0.086849111 container create 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.build-date=20240102, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28264]: 2024-01-19 06:18:06.644764262 +0000 UTC m=+0.017486142 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph:quincy 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28264]: 2024-01-19 06:18:06.883200185 +0000 UTC m=+0.255922064 container init 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, RELEASE=quincy-6441305, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 podman[28264]: 2024-01-19 06:18:06.886249772 +0000 UTC m=+0.258971645 container start 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240102, maintainer=Guillaume Abrioux , RELEASE=quincy-6441305, GIT_CLEAN=True, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: set uid:gid to 167:167 (ceph:ceph) 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable), process ceph-mon, pid 2 2024-01-19T06:18:06.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: pidfile_write: ignore empty --pid-file 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: load: jerasure load: lrc 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: RocksDB version: 6.15.5 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Compile date Jan 9 2024 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: DB SUMMARY 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: DB Session ID: S4Z4ARVX486FC7WWASCG 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: CURRENT file: CURRENT 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: IDENTITY file: IDENTITY 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi134/store.db dir, Total Num: 1, files: 000008.sst 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi134/store.db: 000010.log size: 96792 ; 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.error_if_exists: 0 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.create_if_missing: 0 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.paranoid_checks: 1 2024-01-19T06:18:06.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.env: 0x5620c928fa80 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.fs: Posix File System 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.info_log: 0x5620cb91d000 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_file_opening_threads: 16 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.statistics: (nil) 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.use_fsync: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_log_file_size: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.keep_log_file_num: 1000 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.recycle_log_file_num: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_fallocate: 1 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_mmap_reads: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_mmap_writes: 0 2024-01-19T06:18:06.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.use_direct_reads: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.create_missing_column_families: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.db_log_dir: 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi134/store.db 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.advise_random_on_open: 1 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.db_write_buffer_size: 0 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.write_buffer_manager: 0x5620cc692990 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-19T06:18:06.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.rate_limiter: (nil) 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.wal_recovery_mode: 2 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.enable_thread_tracking: 0 2024-01-19T06:18:06.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.enable_pipelined_write: 0 2024-01-19T06:18:06.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.unordered_write: 0 2024-01-19T06:18:06.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-19T06:18:06.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-19T06:18:06.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.row_cache: None 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.wal_filter: None 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_ingest_behind: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.preserve_deletes: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.two_write_queues: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.manual_wal_flush: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.atomic_flush: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.log_readahead_size: 0 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-19T06:18:06.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.best_efforts_recovery: 0 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.allow_data_in_errors: 0 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.db_host_id: __hostname__ 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_background_jobs: 2 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_background_compactions: -1 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_subcompactions: 1 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_total_wal_size: 0 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-19T06:18:06.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_open_files: -1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bytes_per_sync: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_readahead_size: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_background_flushes: -1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Compression algorithms supported: 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kZSTD supported: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kXpressCompression supported: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kLZ4HCCompression supported: 1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kZlibCompression supported: 1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kSnappyCompression supported: 1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kLZ4Compression supported: 1 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: kBZip2Compression supported: 0 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-19T06:18:06.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000009 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.merge_operator: 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_filter: None 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_filter_factory: None 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.sst_partitioner_factory: None 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5620cb972f90) 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_type: 0 2024-01-19T06:18:06.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_index_type: 0 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_shortening: 1 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: hash_index_allow_collision: 1 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: checksum: 1 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: no_block_cache: 0 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache: 0x5620cb9a5090 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_options: 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: capacity : 536870912 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: num_shard_bits : 4 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: strict_capacity_limit : 0 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_compressed: (nil) 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: persistent_cache: (nil) 2024-01-19T06:18:06.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size: 4096 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size_deviation: 10 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_restart_interval: 16 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_block_restart_interval: 1 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: metadata_block_size: 4096 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: partition_filters: 0 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: use_delta_encoding: 1 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: whole_key_filtering: 1 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: verify_compression: 0 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: format_version: 4 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_index_compression: 1 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_align: 0 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.write_buffer_size: 33554432 2024-01-19T06:18:06.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_write_buffer_number: 2 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression: NoCompression 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression: Disabled 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.prefix_extractor: nullptr 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.num_levels: 7 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-19T06:18:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.level: 32767 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.strategy: 0 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compression_opts.enabled: false 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.target_file_size_base: 67108864 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-19T06:18:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.arena_block_size: 4194304 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.disable_auto_compactions: 0 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-19T06:18:06.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.table_properties_collectors: 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.inplace_update_support: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.bloom_locality: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.max_successive_merges: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.paranoid_file_checks: 0 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.force_consistency_checks: 1 2024-01-19T06:18:06.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.report_bg_io_stats: 0 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.ttl: 2592000 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.enable_blob_files: false 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.min_blob_size: 0 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.blob_file_size: 268435456 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705645086912106, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-01-19T06:18:06.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705645086912858, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92161, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90256, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82995, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705645086, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "bba2f8ef-6423-4ade-92f1-36bc1ab88315", "db_session_id": "S4Z4ARVX486FC7WWASCG"}} 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705645086913234, "job": 1, "event": "recovery_finished"} 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi134/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5620cb939180 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: DB pointer 0x5620cba0e000 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** DB Stats ** 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-19T06:18:06.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: L0 2/0 91.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Sum 2/0 91.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-01-19T06:18:07.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: L0 2/0 91.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Sum 2/0 91.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 172.0 0.00 0.00 1 0.001 0 0 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T06:18:07.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative compaction: 0.00 GB write, 29.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: starting mon.smithi134 rank 0 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi134 fsid 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???) e1 preinit fsid 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).mds e1 new map 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).mds e1 print_map 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: e1 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T06:18:07.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: legacy client fscid: -1 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: No filesystems configured 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 bash[28264]: 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 systemd[1]: Started Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mon.smithi134 is new leader, mons smithi134 in quorum (ranks 0) 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: monmap e1: 1 mons at {smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: fsmap 2024-01-19T06:18:07.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: osdmap e1: 0 total, 0 up, 0 in 2024-01-19T06:18:07.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:06 smithi134 ceph-mon[28283]: mgrmap e1: no daemons active 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.477+0000 7f1062149700 1 Processor -- start 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.477+0000 7f1062149700 1 -- start start 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.477+0000 7f1062149700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.477+0000 7f1062149700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c104310 con 0x7f105c1039c0 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51060/0 (socket says 172.21.15.134:51060) 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105b7fe700 1 -- 172.21.15.134:0/139213149 learned_addr learned my addr 172.21.15.134:0/139213149 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:08.079 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105b7fe700 1 -- 172.21.15.134:0/139213149 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c101fb0 con 0x7f105c1039c0 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105b7fe700 1 --2- 172.21.15.134:0/139213149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f10540098e0 tx=0x7f1054009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c302f554e43c81 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105a7fc700 1 -- 172.21.15.134:0/139213149 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105400b860 con 0x7f105c1039c0 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105a7fc700 1 -- 172.21.15.134:0/139213149 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f105400be30 con 0x7f105c1039c0 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.478+0000 7f105a7fc700 1 -- 172.21.15.134:0/139213149 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1054010380 con 0x7f105c1039c0 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 -- 172.21.15.134:0/139213149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 msgr2=0x7f105c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 --2- 172.21.15.134:0/139213149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f10540098e0 tx=0x7f1054009250 comp rx=0 tx=0).stop 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 -- 172.21.15.134:0/139213149 shutdown_connections 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 --2- 172.21.15.134:0/139213149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c103dd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 -- 172.21.15.134:0/139213149 >> 172.21.15.134:0/139213149 conn(0x7f105c0fda40 msgr2=0x7f105c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 -- 172.21.15.134:0/139213149 shutdown_connections 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 -- 172.21.15.134:0/139213149 wait complete. 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.479+0000 7f1062149700 1 Processor -- start 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1062149700 1 -- start start 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1062149700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:08.080 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1062149700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c111630 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f105b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f105b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51074/0 (socket says 172.21.15.134:51074) 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f105b7fe700 1 -- 172.21.15.134:0/3180554245 learned_addr learned my addr 172.21.15.134:0/3180554245 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f105b7fe700 1 -- 172.21.15.134:0/3180554245 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540094f0 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f105b7fe700 1 --2- 172.21.15.134:0/3180554245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1054000c00 tx=0x7f1054009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1054010a80 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c111830 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f105401a430 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10540245b0 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.480+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c111c40 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.481+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1054024af0 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.481+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f105402eb40 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.481+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c102d60 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.484+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1054029070 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.550+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f105c04bb80 con 0x7f105c1039c0 2024-01-19T06:18:08.081 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.552+0000 7f1043fff700 1 -- 172.21.15.134:0/3180554245 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1054024da0 con 0x7f105c1039c0 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 msgr2=0x7f105c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 --2- 172.21.15.134:0/3180554245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1054000c00 tx=0x7f1054009250 comp rx=0 tx=0).stop 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 shutdown_connections 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 --2- 172.21.15.134:0/3180554245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f105c1039c0 0x7f105c1110f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 >> 172.21.15.134:0/3180554245 conn(0x7f105c0fda40 msgr2=0x7f105c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 shutdown_connections 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:07.553+0000 7f1062149700 1 -- 172.21.15.134:0/3180554245 wait complete. 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote config to /etc/ceph/ceph.conf 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mgr... 2024-01-19T06:18:08.082 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying port 9283 ... 2024-01-19T06:18:08.248 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-573cd99a-b692-11ee-95ae-87774f69a715@mgr.smithi134.qrxokr 2024-01-19T06:18:08.248 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-573cd99a-b692-11ee-95ae-87774f69a715@mgr.smithi134.qrxokr.service: Unit ceph-573cd99a-b692-11ee-95ae-87774f69a715@mgr.smithi134.qrxokr.service not loaded. 2024-01-19T06:18:08.396 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715.target.wants/ceph-573cd99a-b692-11ee-95ae-87774f69a715@mgr.smithi134.qrxokr.service → /etc/systemd/system/ceph-573cd99a-b692-11ee-95ae-87774f69a715@.service. 2024-01-19T06:18:08.891 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:08 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3180554245' entity='client.admin' 2024-01-19T06:18:09.006 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T06:18:09.006 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-01-19T06:18:09.015 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T06:18:09.015 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:18:09.015 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-01-19T06:18:09.015 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-01-19T06:18:09.025 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T06:18:09.025 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr to start... 2024-01-19T06:18:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr... 2024-01-19T06:18:09.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:09 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3831916133' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-19T06:18:10.167 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "573cd99a-b692-11ee-95ae-87774f69a715", 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-19T06:18:10.180 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "smithi134" 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.181 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-19T06:18:10.182 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-19T06:18:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-01-19T06:18:02.238646+0000", 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc980ee700 1 Processor -- start 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc980ee700 1 -- start start 2024-01-19T06:18:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc980ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc980ee700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90102cf0 con 0x7fbc90070fa0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc95e8a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc95e8a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51084/0 (socket says 172.21.15.134:51084) 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.475+0000 7fbc95e8a700 1 -- 172.21.15.134:0/1572645342 learned_addr learned my addr 172.21.15.134:0/1572645342 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.476+0000 7fbc95e8a700 1 -- 172.21.15.134:0/1572645342 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90107840 con 0x7fbc90070fa0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.476+0000 7fbc95e8a700 1 --2- 172.21.15.134:0/1572645342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0098e0 tx=0x7fbc8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f481dfe1a31ee89f server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.476+0000 7fbc94e88700 1 -- 172.21.15.134:0/1572645342 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc8c00b860 con 0x7fbc90070fa0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.476+0000 7fbc94e88700 1 -- 172.21.15.134:0/1572645342 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbc8c00be30 con 0x7fbc90070fa0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.476+0000 7fbc94e88700 1 -- 172.21.15.134:0/1572645342 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc8c010380 con 0x7fbc90070fa0 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 -- 172.21.15.134:0/1572645342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 msgr2=0x7fbc90107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 --2- 172.21.15.134:0/1572645342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0098e0 tx=0x7fbc8c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 -- 172.21.15.134:0/1572645342 shutdown_connections 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 --2- 172.21.15.134:0/1572645342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90107300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 -- 172.21.15.134:0/1572645342 >> 172.21.15.134:0/1572645342 conn(0x7fbc900fc970 msgr2=0x7fbc900fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 -- 172.21.15.134:0/1572645342 shutdown_connections 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.477+0000 7fbc980ee700 1 -- 172.21.15.134:0/1572645342 wait complete. 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc980ee700 1 Processor -- start 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc980ee700 1 -- start start 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc980ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc980ee700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc901138c0 con 0x7fbc90070fa0 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc95e8a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc95e8a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51100/0 (socket says 172.21.15.134:51100) 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc95e8a700 1 -- 172.21.15.134:0/3831916133 learned_addr learned my addr 172.21.15.134:0/3831916133 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc95e8a700 1 -- 172.21.15.134:0/3831916133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c0094f0 con 0x7fbc90070fa0 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc95e8a700 1 --2- 172.21.15.134:0/3831916133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006d40 tx=0x7fbc8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.478+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc8c0109b0 con 0x7fbc90070fa0 2024-01-19T06:18:10.186 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbc8c024440 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc90113ac0 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc8c01a5f0 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc90113ed0 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbc8c01aaf0 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.479+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbc8c02da60 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.480+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9004bb80 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.482+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbc8c024970 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.552+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbc90108500 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.553+0000 7fbc867fc700 1 -- 172.21.15.134:0/3831916133 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fbc8c0245b0 con 0x7fbc90070fa0 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 msgr2=0x7fbc90113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 --2- 172.21.15.134:0/3831916133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006d40 tx=0x7fbc8c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 shutdown_connections 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 --2- 172.21.15.134:0/3831916133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90070fa0 0x7fbc90113380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 >> 172.21.15.134:0/3831916133 conn(0x7fbc900fc970 msgr2=0x7fbc900fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:10.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.554+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 shutdown_connections 2024-01-19T06:18:10.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:09.555+0000 7fbc980ee700 1 -- 172.21.15.134:0/3831916133 wait complete. 2024-01-19T06:18:10.188 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (1/15)... 2024-01-19T06:18:13.036 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:12 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3157391005' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "573cd99a-b692-11ee-95ae-87774f69a715", 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-19T06:18:13.267 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "smithi134" 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-19T06:18:13.268 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.269 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-19T06:18:13.269 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:13.269 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-19T06:18:13.269 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-19T06:18:13.269 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-19T06:18:13.276 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-19T06:18:13.276 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-19T06:18:13.277 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-19T06:18:13.278 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-01-19T06:18:02.238646+0000", 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1ea14a7700 1 Processor -- start 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1ea14a7700 1 -- start start 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1ea14a7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1ea14a7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c102ce0 con 0x7f1e9c076c00 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1e9affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.633+0000 7f1e9affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51110/0 (socket says 172.21.15.134:51110) 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e9affd700 1 -- 172.21.15.134:0/3753439062 learned_addr learned my addr 172.21.15.134:0/3753439062 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e9affd700 1 -- 172.21.15.134:0/3753439062 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c107830 con 0x7f1e9c076c00 2024-01-19T06:18:13.279 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e9affd700 1 --2- 172.21.15.134:0/3753439062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e900098e0 tx=0x7f1e90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298a40b24112775c server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e99ffb700 1 -- 172.21.15.134:0/3753439062 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e9000b860 con 0x7f1e9c076c00 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e99ffb700 1 -- 172.21.15.134:0/3753439062 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1e9000be30 con 0x7f1e9c076c00 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.634+0000 7f1e99ffb700 1 -- 172.21.15.134:0/3753439062 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e90010380 con 0x7f1e9c076c00 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3753439062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 msgr2=0x7f1e9c1072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 --2- 172.21.15.134:0/3753439062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e900098e0 tx=0x7f1e90009250 comp rx=0 tx=0).stop 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3753439062 shutdown_connections 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 --2- 172.21.15.134:0/3753439062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c1072f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3753439062 >> 172.21.15.134:0/3753439062 conn(0x7f1e9c0fc980 msgr2=0x7f1e9c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3753439062 shutdown_connections 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.635+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3753439062 wait complete. 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1ea14a7700 1 Processor -- start 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1ea14a7700 1 -- start start 2024-01-19T06:18:13.280 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1ea14a7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1ea14a7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c1138b0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1e9affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1e9affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51114/0 (socket says 172.21.15.134:51114) 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1e9affd700 1 -- 172.21.15.134:0/3157391005 learned_addr learned my addr 172.21.15.134:0/3157391005 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1e9affd700 1 -- 172.21.15.134:0/3157391005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e900094f0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.636+0000 7f1e9affd700 1 --2- 172.21.15.134:0/3157391005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e90006b00 tx=0x7f1e90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e90010a80 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1e9001a430 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c113ab0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e900245b0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c113ec0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1e90024ab0 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.637+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1e9002eb00 con 0x7f1e9c076c00 2024-01-19T06:18:13.281 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.638+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c04bb80 con 0x7f1e9c076c00 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.640+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1e9002e440 con 0x7f1e9c076c00 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.713+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1e9c10ce20 con 0x7f1e9c076c00 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.713+0000 7f1e837fe700 1 -- 172.21.15.134:0/3157391005 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f1e9002ed60 con 0x7f1e9c076c00 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 msgr2=0x7f1e9c113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 --2- 172.21.15.134:0/3157391005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e90006b00 tx=0x7f1e90009250 comp rx=0 tx=0).stop 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 shutdown_connections 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 --2- 172.21.15.134:0/3157391005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e9c076c00 0x7f1e9c113370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 >> 172.21.15.134:0/3157391005 conn(0x7f1e9c0fc980 msgr2=0x7f1e9c0fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 shutdown_connections 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:12.715+0000 7f1ea14a7700 1 -- 172.21.15.134:0/3157391005 wait complete. 2024-01-19T06:18:13.282 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (2/15)... 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: Activating manager daemon smithi134.qrxokr 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: mgrmap e2: smithi134.qrxokr(active, starting, since 0.0028488s) 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.qrxokr", "id": "smithi134.qrxokr"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: Manager daemon smithi134.qrxokr is now available 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/mirror_snapshot_schedule"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/trash_purge_schedule"}]: dispatch 2024-01-19T06:18:15.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:15.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:15.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:15 smithi134 ceph-mon[28283]: from='mgr.14100 172.21.15.134:0/3277785554' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:17.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:17 smithi134 ceph-mon[28283]: mgrmap e3: smithi134.qrxokr(active, since 1.00697s) 2024-01-19T06:18:17.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:17 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2866004162' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-19T06:18:19.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:19 smithi134 ceph-mon[28283]: mgrmap e4: smithi134.qrxokr(active, since 3s) 2024-01-19T06:18:23.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:23.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:23.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "573cd99a-b692-11ee-95ae-87774f69a715", 2024-01-19T06:18:23.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-01-19T06:18:23.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "smithi134" 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-19T06:18:23.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-19T06:18:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-19T06:18:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-19T06:18:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-19T06:18:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-19T06:18:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.160 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-19T06:18:23.161 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-01-19T06:18:02.238646+0000", 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T06:18:23.162 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.542+0000 7ff888267700 1 Processor -- start 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.542+0000 7ff888267700 1 -- start start 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.542+0000 7ff888267700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.542+0000 7ff888267700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880103100 con 0x7ff8801027b0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.542+0000 7ff886003700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff886003700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43312/0 (socket says 172.21.15.134:43312) 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff886003700 1 -- 172.21.15.134:0/2130935783 learned_addr learned my addr 172.21.15.134:0/2130935783 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff886003700 1 -- 172.21.15.134:0/2130935783 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880103240 con 0x7ff8801027b0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff886003700 1 --2- 172.21.15.134:0/2130935783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0098e0 tx=0x7ff87c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f350f786c7da69 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff885001700 1 -- 172.21.15.134:0/2130935783 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff87c00b860 con 0x7ff8801027b0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff885001700 1 -- 172.21.15.134:0/2130935783 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff87c00be30 con 0x7ff8801027b0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.543+0000 7ff885001700 1 -- 172.21.15.134:0/2130935783 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff87c010380 con 0x7ff8801027b0 2024-01-19T06:18:23.163 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 -- 172.21.15.134:0/2130935783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 msgr2=0x7ff880102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 --2- 172.21.15.134:0/2130935783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0098e0 tx=0x7ff87c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 -- 172.21.15.134:0/2130935783 shutdown_connections 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 --2- 172.21.15.134:0/2130935783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880102bc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 -- 172.21.15.134:0/2130935783 >> 172.21.15.134:0/2130935783 conn(0x7ff8800fc0b0 msgr2=0x7ff8800fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 -- 172.21.15.134:0/2130935783 shutdown_connections 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.544+0000 7ff888267700 1 -- 172.21.15.134:0/2130935783 wait complete. 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff888267700 1 Processor -- start 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff888267700 1 -- start start 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff888267700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff888267700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880111690 con 0x7ff8801027b0 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff886003700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff886003700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43320/0 (socket says 172.21.15.134:43320) 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff886003700 1 -- 172.21.15.134:0/2866004162 learned_addr learned my addr 172.21.15.134:0/2866004162 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:23.164 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff886003700 1 -- 172.21.15.134:0/2866004162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0094f0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff886003700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff87c000c00 tx=0x7ff87c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.545+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff87c010a80 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff87c01a430 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff880111890 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff87c0245b0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff880111ca0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 3) v1 ==== 43164+0+0 (secure 0 0 0) 0x7ff87c0247a0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.546+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88004bb80 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.547+0000 7ff876ffd700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 0x7ff86c038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.547+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff87c04a8a0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.547+0000 7ff885802700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 0x7ff86c038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.547+0000 7ff885802700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 0x7ff86c038970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff870009990 tx=0x7ff870006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.552+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff87c023bc0 con 0x7ff8801027b0 2024-01-19T06:18:23.165 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.815+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff88010b070 con 0x7ff8801027b0 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.815+0000 7ff876ffd700 1 -- 172.21.15.134:0/2866004162 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7ff87c023390 con 0x7ff8801027b0 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 msgr2=0x7ff86c038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 0x7ff86c038970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff870009990 tx=0x7ff870006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 msgr2=0x7ff880111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff87c000c00 tx=0x7ff87c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 shutdown_connections 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7ff86c0364c0 0x7ff86c038970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 --2- 172.21.15.134:0/2866004162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8801027b0 0x7ff880111150 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 >> 172.21.15.134:0/2866004162 conn(0x7ff8800fc0b0 msgr2=0x7ff8800fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 shutdown_connections 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:16.817+0000 7ff888267700 1 -- 172.21.15.134:0/2866004162 wait complete. 2024-01-19T06:18:23.166 INFO:teuthology.orchestra.run.smithi134.stdout:mgr is available 2024-01-19T06:18:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:25 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1775940346' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-19T06:18:25.953 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-19T06:18:25.960 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f0530f95700 1 Processor -- start 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f0530f95700 1 -- start start 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f0530f95700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:25.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f0530f95700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c104310 con 0x7f052c1039c0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f052a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f052a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43322/0 (socket says 172.21.15.134:43322) 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.173+0000 7f052a59c700 1 -- 172.21.15.134:0/431510497 learned_addr learned my addr 172.21.15.134:0/431510497 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f052a59c700 1 -- 172.21.15.134:0/431510497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c101fb0 con 0x7f052c1039c0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f052a59c700 1 --2- 172.21.15.134:0/431510497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbdb2c3f3502badd server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f052959a700 1 -- 172.21.15.134:0/431510497 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c00b860 con 0x7f052c1039c0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f052959a700 1 -- 172.21.15.134:0/431510497 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f051c00be30 con 0x7f052c1039c0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f052959a700 1 -- 172.21.15.134:0/431510497 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c010380 con 0x7f052c1039c0 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f0530f95700 1 -- 172.21.15.134:0/431510497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 msgr2=0x7f052c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.174+0000 7f0530f95700 1 --2- 172.21.15.134:0/431510497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f051c0098e0 tx=0x7f051c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- 172.21.15.134:0/431510497 shutdown_connections 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 --2- 172.21.15.134:0/431510497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c103dd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- 172.21.15.134:0/431510497 >> 172.21.15.134:0/431510497 conn(0x7f052c0fda40 msgr2=0x7f052c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:25.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- 172.21.15.134:0/431510497 shutdown_connections 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- 172.21.15.134:0/431510497 wait complete. 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 Processor -- start 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- start start 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f0530f95700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c111630 con 0x7f052c1039c0 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f052a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f052a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43326/0 (socket says 172.21.15.134:43326) 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.175+0000 7f052a59c700 1 -- 172.21.15.134:0/1775940346 learned_addr learned my addr 172.21.15.134:0/1775940346 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:25.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f052a59c700 1 -- 172.21.15.134:0/1775940346 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0094f0 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f052a59c700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f051c006b00 tx=0x7f051c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c0109b0 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f051c01a430 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c111830 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f051c0245b0 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.176+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c111c40 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f051c024830 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f051affd700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f051c04ab70 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c102d60 con 0x7f052c1039c0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f0529d9b700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:25.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.177+0000 7f0529d9b700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 0x7f0514038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0520009990 tx=0x7f0520006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.182+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f051c018350 con 0x7f052c1039c0 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.360+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f052c04bb80 con 0x7f052c1039c0 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.361+0000 7f051affd700 1 -- 172.21.15.134:0/1775940346 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f051c048020 con 0x7f052c1039c0 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 msgr2=0x7f0514038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 0x7f0514038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0520009990 tx=0x7f0520006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 msgr2=0x7f052c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f051c006b00 tx=0x7f051c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 shutdown_connections 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f05140365a0 0x7f0514038a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 --2- 172.21.15.134:0/1775940346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f052c1039c0 0x7f052c1110f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 >> 172.21.15.134:0/1775940346 conn(0x7f052c0fda40 msgr2=0x7f052c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 shutdown_connections 2024-01-19T06:18:25.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:25.364+0000 7f0530f95700 1 -- 172.21.15.134:0/1775940346 wait complete. 2024-01-19T06:18:25.966 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling cephadm module... 2024-01-19T06:18:26.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:26 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/4022472801' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f574153c700 1 Processor -- start 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f574153c700 1 -- start start 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f574153c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f574153c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c104300 con 0x7f573c1039b0 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f573affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f573affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43340/0 (socket says 172.21.15.134:43340) 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f573affd700 1 -- 172.21.15.134:0/1696078757 learned_addr learned my addr 172.21.15.134:0/1696078757 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:27.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.389+0000 7f573affd700 1 -- 172.21.15.134:0/1696078757 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c104440 con 0x7f573c1039b0 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f573affd700 1 --2- 172.21.15.134:0/1696078757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc46d9d0ae171b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f5739ffb700 1 -- 172.21.15.134:0/1696078757 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c00b860 con 0x7f573c1039b0 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f5739ffb700 1 -- 172.21.15.134:0/1696078757 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f572c00be30 con 0x7f573c1039b0 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f5739ffb700 1 -- 172.21.15.134:0/1696078757 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c010380 con 0x7f573c1039b0 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 -- 172.21.15.134:0/1696078757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 msgr2=0x7f573c103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 --2- 172.21.15.134:0/1696078757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 -- 172.21.15.134:0/1696078757 shutdown_connections 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 --2- 172.21.15.134:0/1696078757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c103dc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 -- 172.21.15.134:0/1696078757 >> 172.21.15.134:0/1696078757 conn(0x7f573c0fc0b0 msgr2=0x7f573c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 -- 172.21.15.134:0/1696078757 shutdown_connections 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.390+0000 7f574153c700 1 -- 172.21.15.134:0/1696078757 wait complete. 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f574153c700 1 Processor -- start 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f574153c700 1 -- start start 2024-01-19T06:18:27.307 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f574153c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f574153c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c111690 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f573affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f573affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43350/0 (socket says 172.21.15.134:43350) 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.391+0000 7f573affd700 1 -- 172.21.15.134:0/4022472801 learned_addr learned my addr 172.21.15.134:0/4022472801 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f573affd700 1 -- 172.21.15.134:0/4022472801 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0094f0 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f573affd700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f572c006b00 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c0109b0 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f572c024440 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c111890 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c01a5f0 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.392+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c111ca0 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f572c01a8b0 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f57237fe700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 0x7f5724038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c04bb80 con 0x7f573c1039b0 2024-01-19T06:18:27.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f572c04a580 con 0x7f573c1039b0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f573a7fc700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 0x7f5724038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.393+0000 7f573a7fc700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 0x7f5724038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5730009990 tx=0x7f5730006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.398+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f572c01ab60 con 0x7f573c1039b0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.637+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f573c10af90 con 0x7f573c1039b0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.689+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f572c00fdc0 con 0x7f573c1039b0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.690+0000 7f57237fe700 1 -- 172.21.15.134:0/4022472801 <== mon.0 v2:172.21.15.134:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f572c018b40 con 0x7f573c1039b0 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 msgr2=0x7f5724038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 0x7f5724038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5730009990 tx=0x7f5730006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 msgr2=0x7f573c111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f572c006b00 tx=0x7f572c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 shutdown_connections 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f57240365a0 0x7f5724038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 --2- 172.21.15.134:0/4022472801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f573c1039b0 0x7f573c111150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 >> 172.21.15.134:0/4022472801 conn(0x7f573c0fc0b0 msgr2=0x7f573c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 shutdown_connections 2024-01-19T06:18:27.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:26.693+0000 7f574153c700 1 -- 172.21.15.134:0/4022472801 wait complete. 2024-01-19T06:18:27.872 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:27 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/4022472801' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-19T06:18:27.872 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:27 smithi134 ceph-mon[28283]: mgrmap e5: smithi134.qrxokr(active, since 11s) 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "smithi134.qrxokr", 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-19T06:18:28.506 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1ce6061700 1 Processor -- start 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1ce6061700 1 -- start start 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1ce6061700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1ce6061700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0104240 con 0x7f1ce01038f0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1cdf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1cdf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48168/0 (socket says 172.21.15.134:48168) 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1cdf7fe700 1 -- 172.21.15.134:0/124377639 learned_addr learned my addr 172.21.15.134:0/124377639 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.667+0000 7f1cdf7fe700 1 -- 172.21.15.134:0/124377639 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce0104380 con 0x7f1ce01038f0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1cdf7fe700 1 --2- 172.21.15.134:0/124377639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00098e0 tx=0x7f1cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3562b1d5ef88a3eb server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1cde7fc700 1 -- 172.21.15.134:0/124377639 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd000b860 con 0x7f1ce01038f0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1cde7fc700 1 -- 172.21.15.134:0/124377639 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1cd000be30 con 0x7f1ce01038f0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1cde7fc700 1 -- 172.21.15.134:0/124377639 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd0010380 con 0x7f1ce01038f0 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1ce6061700 1 -- 172.21.15.134:0/124377639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 msgr2=0x7f1ce0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:28.507 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.668+0000 7f1ce6061700 1 --2- 172.21.15.134:0/124377639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00098e0 tx=0x7f1cd0009250 comp rx=0 tx=0).stop 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- 172.21.15.134:0/124377639 shutdown_connections 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 --2- 172.21.15.134:0/124377639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0103d00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- 172.21.15.134:0/124377639 >> 172.21.15.134:0/124377639 conn(0x7f1ce00fc0b0 msgr2=0x7f1ce00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- 172.21.15.134:0/124377639 shutdown_connections 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- 172.21.15.134:0/124377639 wait complete. 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 Processor -- start 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- start start 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.669+0000 7f1ce6061700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0111650 con 0x7f1ce01038f0 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cdf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cdf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48176/0 (socket says 172.21.15.134:48176) 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cdf7fe700 1 -- 172.21.15.134:0/2651814193 learned_addr learned my addr 172.21.15.134:0/2651814193 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cdf7fe700 1 -- 172.21.15.134:0/2651814193 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd00094f0 con 0x7f1ce01038f0 2024-01-19T06:18:28.508 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cdf7fe700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000c00 tx=0x7f1cd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd0010a80 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1cd001a430 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce0111850 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd00245b0 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.670+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce0111c60 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f1cd00247a0 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1cc7fff700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 0x7f1cc8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1cd004ba90 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1cdeffd700 1 -- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 msgr2=0x7f1cc8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1cdeffd700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 0x7f1cc8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.671+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce004bb80 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.676+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1cd0018350 con 0x7f1ce01038f0 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.872+0000 7f1cdeffd700 1 -- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 msgr2=0x7f1cc8038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:28.509 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.872+0000 7f1cdeffd700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 0x7f1cc8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.920+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1ce010ac80 con 0x7f1ce01038f0 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.920+0000 7f1cc7fff700 1 -- 172.21.15.134:0/2651814193 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1cd0023b80 con 0x7f1ce01038f0 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 msgr2=0x7f1cc8038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 0x7f1cc8038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 msgr2=0x7f1ce0111110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000c00 tx=0x7f1cd0009250 comp rx=0 tx=0).stop 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 shutdown_connections 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7f1cc8036640 0x7f1cc8038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 --2- 172.21.15.134:0/2651814193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ce01038f0 0x7f1ce0111110 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 >> 172.21.15.134:0/2651814193 conn(0x7f1ce00fc0b0 msgr2=0x7f1ce00fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 shutdown_connections 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:27.922+0000 7f1ce6061700 1 -- 172.21.15.134:0/2651814193 wait complete. 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-01-19T06:18:28.510 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 5... 2024-01-19T06:18:28.799 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:28 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2651814193' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-19T06:18:33.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: Active manager daemon smithi134.qrxokr restarted 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: Activating manager daemon smithi134.qrxokr 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: osdmap e2: 0 total, 0 up, 0 in 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: mgrmap e6: smithi134.qrxokr(active, starting, since 0.00313104s) 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.qrxokr", "id": "smithi134.qrxokr"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: Manager daemon smithi134.qrxokr is now available 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:33.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:33 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:34.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: Found migration_current of "None". Setting to last migration. 2024-01-19T06:18:34.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/mirror_snapshot_schedule"}]: dispatch 2024-01-19T06:18:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/trash_purge_schedule"}]: dispatch 2024-01-19T06:18:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:34 smithi134 ceph-mon[28283]: mgrmap e7: smithi134.qrxokr(active, since 1.00812s) 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda4a382700 1 Processor -- start 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda4a382700 1 -- start start 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda4a382700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda4a382700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda44102c20 con 0x7fda441022d0 2024-01-19T06:18:34.767 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda43fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda43fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48190/0 (socket says 172.21.15.134:48190) 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.916+0000 7fda43fff700 1 -- 172.21.15.134:0/2182908855 learned_addr learned my addr 172.21.15.134:0/2182908855 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.917+0000 7fda43fff700 1 -- 172.21.15.134:0/2182908855 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda44102d60 con 0x7fda441022d0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.917+0000 7fda43fff700 1 --2- 172.21.15.134:0/2182908855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda380098e0 tx=0x7fda38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41c478b68cb84b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.917+0000 7fda42ffd700 1 -- 172.21.15.134:0/2182908855 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda3800b860 con 0x7fda441022d0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.917+0000 7fda42ffd700 1 -- 172.21.15.134:0/2182908855 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fda3800be30 con 0x7fda441022d0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.917+0000 7fda42ffd700 1 -- 172.21.15.134:0/2182908855 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda38010380 con 0x7fda441022d0 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- 172.21.15.134:0/2182908855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 msgr2=0x7fda441026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 --2- 172.21.15.134:0/2182908855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fda380098e0 tx=0x7fda38009250 comp rx=0 tx=0).stop 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- 172.21.15.134:0/2182908855 shutdown_connections 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 --2- 172.21.15.134:0/2182908855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda441026e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- 172.21.15.134:0/2182908855 >> 172.21.15.134:0/2182908855 conn(0x7fda440fda50 msgr2=0x7fda440ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- 172.21.15.134:0/2182908855 shutdown_connections 2024-01-19T06:18:34.768 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- 172.21.15.134:0/2182908855 wait complete. 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 Processor -- start 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.918+0000 7fda4a382700 1 -- start start 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda4a382700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda4a382700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda44111680 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda43fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda43fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48196/0 (socket says 172.21.15.134:48196) 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda43fff700 1 -- 172.21.15.134:0/2418477960 learned_addr learned my addr 172.21.15.134:0/2418477960 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda43fff700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda380094f0 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda43fff700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda38000c00 tx=0x7fda38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda38010a80 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda44111880 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fda3801a430 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda380245b0 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.919+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda44111c90 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.920+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fda380247f0 con 0x7fda441022d0 2024-01-19T06:18:34.769 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.921+0000 7fda40ff9700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.921+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda2c0391b0 con 0x7fda2c0365f0 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.921+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda3804acf0 con 0x7fda441022d0 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.921+0000 7fda437fe700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:28.921+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:29.122+0000 7fda437fe700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:29.122+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:29.523+0000 7fda437fe700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:29.523+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:30.324+0000 7fda437fe700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:30.324+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:31.925+0000 7fda437fe700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/2771969091 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:31.925+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:33.203+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7fda3801a610 con 0x7fda441022d0 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:33.204+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 msgr2=0x7fda2c038aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:18:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:33.204+0000 7fda40ff9700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.207+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fda3804c790 con 0x7fda441022d0 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.208+0000 7fda40ff9700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 0x7fda2c03c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.208+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda2c0391b0 con 0x7fda2c039f60 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.208+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 0x7fda2c03c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.209+0000 7fda437fe700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 0x7fda2c03c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda340039d0 tx=0x7fda34009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.209+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fda2c0391b0 con 0x7fda2c039f60 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.215+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fda4404bb80 con 0x7fda2c039f60 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.215+0000 7fda40ff9700 1 -- 172.21.15.134:0/2418477960 <== mgr.14118 v2:172.21.15.134:6800/4139544225 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fda4404bb80 con 0x7fda2c039f60 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 msgr2=0x7fda2c03c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 0x7fda2c03c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda340039d0 tx=0x7fda34009350 comp rx=0 tx=0).stop 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 msgr2=0x7fda44111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda38000c00 tx=0x7fda38009250 comp rx=0 tx=0).stop 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 shutdown_connections 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fda2c039f60 0x7fda2c03c340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda441022d0 0x7fda44111140 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:34.771 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 --2- 172.21.15.134:0/2418477960 >> [v2:172.21.15.134:6800/2771969091,v1:172.21.15.134:6801/2771969091] conn(0x7fda2c0365f0 0x7fda2c038aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:34.772 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 >> 172.21.15.134:0/2418477960 conn(0x7fda440fda50 msgr2=0x7fda440ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:34.772 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 shutdown_connections 2024-01-19T06:18:34.772 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:34.216+0000 7fda4a382700 1 -- 172.21.15.134:0/2418477960 wait complete. 2024-01-19T06:18:34.772 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 5 is available 2024-01-19T06:18:34.772 INFO:teuthology.orchestra.run.smithi134.stdout:Setting orchestrator backend to cephadm... 2024-01-19T06:18:35.630 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:35 smithi134 ceph-mon[28283]: [19/Jan/2024:06:18:33] ENGINE Bus STARTING 2024-01-19T06:18:35.630 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:35 smithi134 ceph-mon[28283]: [19/Jan/2024:06:18:34] ENGINE Serving on https://172.21.15.134:7150 2024-01-19T06:18:35.630 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:35 smithi134 ceph-mon[28283]: [19/Jan/2024:06:18:34] ENGINE Bus STARTED 2024-01-19T06:18:35.630 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:35 smithi134 ceph-mon[28283]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-19T06:18:35.630 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:35 smithi134 ceph-mon[28283]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b93fc700 1 Processor -- start 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b93fc700 1 -- start start 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b93fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b93fc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4105420 con 0x7f85b4076c20 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48274/0 (socket says 172.21.15.134:48274) 2024-01-19T06:18:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b2ffd700 1 -- 172.21.15.134:0/613528870 learned_addr learned my addr 172.21.15.134:0/613528870 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b2ffd700 1 -- 172.21.15.134:0/613528870 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4105560 con 0x7f85b4076c20 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.182+0000 7f85b2ffd700 1 --2- 172.21.15.134:0/613528870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f85a40098e0 tx=0x7f85a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5932f56adec96f server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b1ffb700 1 -- 172.21.15.134:0/613528870 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a400b860 con 0x7f85b4076c20 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b1ffb700 1 -- 172.21.15.134:0/613528870 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f85a400be30 con 0x7f85b4076c20 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b1ffb700 1 -- 172.21.15.134:0/613528870 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4010380 con 0x7f85b4076c20 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 -- 172.21.15.134:0/613528870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 msgr2=0x7f85b4104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 --2- 172.21.15.134:0/613528870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f85a40098e0 tx=0x7f85a4009250 comp rx=0 tx=0).stop 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 -- 172.21.15.134:0/613528870 shutdown_connections 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 --2- 172.21.15.134:0/613528870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4104ee0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 -- 172.21.15.134:0/613528870 >> 172.21.15.134:0/613528870 conn(0x7f85b40fc950 msgr2=0x7f85b40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 -- 172.21.15.134:0/613528870 shutdown_connections 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.183+0000 7f85b93fc700 1 -- 172.21.15.134:0/613528870 wait complete. 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b93fc700 1 Processor -- start 2024-01-19T06:18:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b93fc700 1 -- start start 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b93fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b93fc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b4111690 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48284/0 (socket says 172.21.15.134:48284) 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b2ffd700 1 -- 172.21.15.134:0/3633234724 learned_addr learned my addr 172.21.15.134:0/3633234724 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b2ffd700 1 -- 172.21.15.134:0/3633234724 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40094f0 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.184+0000 7f85b2ffd700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f85a4006b00 tx=0x7f85a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a4010a80 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f85a401a430 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b4111890 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85a40245b0 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b4111ca0 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.185+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f85a4024830 con 0x7f85b4076c20 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.186+0000 7f859b7fe700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 0x7f859c038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.186+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f85a404a9f0 con 0x7f85b4076c20 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.186+0000 7f85b27fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 0x7f859c038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.186+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b404bb80 con 0x7f85b4076c20 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.186+0000 7f85b27fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 0x7f859c038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009990 tx=0x7f85a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.191+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f85a4018350 con 0x7f85b4076c20 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.255+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f85a402e960 con 0x7f85b4076c20 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.392+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f85b410ad10 con 0x7f859c0364c0 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.401+0000 7f859b7fe700 1 -- 172.21.15.134:0/3633234724 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f85b410ad10 con 0x7f859c0364c0 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 msgr2=0x7f859c038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 0x7f859c038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009990 tx=0x7f85a8006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 msgr2=0x7f85b4111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f85a4006b00 tx=0x7f85a4009250 comp rx=0 tx=0).stop 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 shutdown_connections 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f859c0364c0 0x7f859c038970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 --2- 172.21.15.134:0/3633234724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85b4076c20 0x7f85b4111150 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:35.908 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 >> 172.21.15.134:0/3633234724 conn(0x7f85b40fc950 msgr2=0x7f85b40fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:35.909 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 shutdown_connections 2024-01-19T06:18:35.909 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:35.404+0000 7f85b93fc700 1 -- 172.21.15.134:0/3633234724 wait complete. 2024-01-19T06:18:36.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:36 smithi134 ceph-mon[28283]: mgrmap e8: smithi134.qrxokr(active, since 2s) 2024-01-19T06:18:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:36 smithi134 ceph-mon[28283]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:36 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:36 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:37.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout value unchanged 2024-01-19T06:18:37.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.508+0000 7f54297a1700 1 Processor -- start 2024-01-19T06:18:37.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.508+0000 7f54297a1700 1 -- start start 2024-01-19T06:18:37.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.508+0000 7f54297a1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.508+0000 7f54297a1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424105420 con 0x7f5424070fc0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.508+0000 7f5422ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f5422ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58828/0 (socket says 172.21.15.134:58828) 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f5422ffd700 1 -- 172.21.15.134:0/3273145666 learned_addr learned my addr 172.21.15.134:0/3273145666 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f5422ffd700 1 -- 172.21.15.134:0/3273145666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424105560 con 0x7f5424070fc0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f5422ffd700 1 --2- 172.21.15.134:0/3273145666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5414009e50 tx=0x7f5414009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67e66c90f25ed32 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f54227fc700 1 -- 172.21.15.134:0/3273145666 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54140092e0 con 0x7f5424070fc0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f54227fc700 1 -- 172.21.15.134:0/3273145666 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f541400be90 con 0x7f5424070fc0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.509+0000 7f54227fc700 1 -- 172.21.15.134:0/3273145666 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5414007400 con 0x7f5424070fc0 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 -- 172.21.15.134:0/3273145666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 msgr2=0x7f5424104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 --2- 172.21.15.134:0/3273145666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5414009e50 tx=0x7f5414009840 comp rx=0 tx=0).stop 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 -- 172.21.15.134:0/3273145666 shutdown_connections 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 --2- 172.21.15.134:0/3273145666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f5424104ee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 -- 172.21.15.134:0/3273145666 >> 172.21.15.134:0/3273145666 conn(0x7f54240fc970 msgr2=0x7f54240fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:37.255 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 -- 172.21.15.134:0/3273145666 shutdown_connections 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 -- 172.21.15.134:0/3273145666 wait complete. 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.510+0000 7f54297a1700 1 Processor -- start 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f54297a1700 1 -- start start 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f54297a1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f54297a1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241116f0 con 0x7f5424070fc0 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f5422ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f5422ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58832/0 (socket says 172.21.15.134:58832) 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f5422ffd700 1 -- 172.21.15.134:0/1847473403 learned_addr learned my addr 172.21.15.134:0/1847473403 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f5422ffd700 1 -- 172.21.15.134:0/1847473403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009ae0 con 0x7f5424070fc0 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f5422ffd700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5414009fd0 tx=0x7f5414009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:37.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.511+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5414007670 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5414007c40 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54241118f0 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f541401cb60 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5424112590 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5414019070 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f541bfff700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 0x7f5410038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5414049ec0 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.512+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424111d60 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.513+0000 7f541a7fc700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 0x7f5410038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.513+0000 7f541a7fc700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 0x7f5410038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f540c009990 tx=0x7f540c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.517+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5414017080 con 0x7f5424070fc0 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.701+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f5424112890 con 0x7f5410036640 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.701+0000 7f541bfff700 1 -- 172.21.15.134:0/1847473403 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f5424112890 con 0x7f5410036640 2024-01-19T06:18:37.257 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 msgr2=0x7f5410038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 0x7f5410038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f540c009990 tx=0x7f540c006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 msgr2=0x7f54241111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5414009fd0 tx=0x7f5414009840 comp rx=0 tx=0).stop 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 shutdown_connections 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5410036640 0x7f5410038af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 --2- 172.21.15.134:0/1847473403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424070fc0 0x7f54241111b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.703+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 >> 172.21.15.134:0/1847473403 conn(0x7f54240fc970 msgr2=0x7f54240fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.704+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 shutdown_connections 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:36.704+0000 7f54297a1700 1 -- 172.21.15.134:0/1847473403 wait complete. 2024-01-19T06:18:37.258 INFO:teuthology.orchestra.run.smithi134.stdout:Generating ssh key... 2024-01-19T06:18:37.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:37 smithi134 ceph-mon[28283]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4fe0f0700 1 Processor -- start 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4fe0f0700 1 -- start start 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4fe0f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4fe0f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8104310 con 0x7fd4f81039c0 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4f77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:38.486 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4f77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58844/0 (socket says 172.21.15.134:58844) 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4f77fe700 1 -- 172.21.15.134:0/727542261 learned_addr learned my addr 172.21.15.134:0/727542261 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4f77fe700 1 -- 172.21.15.134:0/727542261 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8101fb0 con 0x7fd4f81039c0 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.636+0000 7fd4f77fe700 1 --2- 172.21.15.134:0/727542261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80098e0 tx=0x7fd4e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f7a06606aa4fb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4f67fc700 1 -- 172.21.15.134:0/727542261 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e800b860 con 0x7fd4f81039c0 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4f67fc700 1 -- 172.21.15.134:0/727542261 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd4e800be30 con 0x7fd4f81039c0 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4f67fc700 1 -- 172.21.15.134:0/727542261 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e8010380 con 0x7fd4f81039c0 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/727542261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 msgr2=0x7fd4f8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/727542261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80098e0 tx=0x7fd4e8009250 comp rx=0 tx=0).stop 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/727542261 shutdown_connections 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/727542261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f8103dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/727542261 >> 172.21.15.134:0/727542261 conn(0x7fd4f80fda40 msgr2=0x7fd4f80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/727542261 shutdown_connections 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.637+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/727542261 wait complete. 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4fe0f0700 1 Processor -- start 2024-01-19T06:18:38.487 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4fe0f0700 1 -- start start 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4fe0f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4fe0f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8111630 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4f77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4f77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58848/0 (socket says 172.21.15.134:58848) 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4f77fe700 1 -- 172.21.15.134:0/1622169102 learned_addr learned my addr 172.21.15.134:0/1622169102 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.638+0000 7fd4f77fe700 1 -- 172.21.15.134:0/1622169102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80094f0 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4f77fe700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8006b00 tx=0x7fd4e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e80109b0 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f8111830 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd4e801a430 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e80245b0 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.639+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f8111c40 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd4e8024830 con 0x7fd4f81039c0 2024-01-19T06:18:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4dffff700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 0x7fd4e0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd4e804ac00 con 0x7fd4f81039c0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8102d60 con 0x7fd4f81039c0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4f6ffd700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 0x7fd4e0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.640+0000 7fd4f6ffd700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 0x7fd4e0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009990 tx=0x7fd4ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.645+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd4e8018350 con 0x7fd4f81039c0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.829+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4f810adc0 con 0x7fd4e00365f0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.919+0000 7fd4dffff700 1 -- 172.21.15.134:0/1622169102 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd4f810adc0 con 0x7fd4e00365f0 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.921+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 msgr2=0x7fd4e0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.921+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 0x7fd4e0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009990 tx=0x7fd4ec006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.921+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 msgr2=0x7fd4f81110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.921+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8006b00 tx=0x7fd4e8009250 comp rx=0 tx=0).stop 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 shutdown_connections 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd4e00365f0 0x7fd4e0038aa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:38.489 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 --2- 172.21.15.134:0/1622169102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4f81039c0 0x7fd4f81110f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:38.490 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 >> 172.21.15.134:0/1622169102 conn(0x7fd4f80fda40 msgr2=0x7fd4f80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:38.490 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 shutdown_connections 2024-01-19T06:18:38.490 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:37.922+0000 7fd4fe0f0700 1 -- 172.21.15.134:0/1622169102 wait complete. 2024-01-19T06:18:39.068 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:38 smithi134 ceph-mon[28283]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:39.068 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:38 smithi134 ceph-mon[28283]: Generating ssh key... 2024-01-19T06:18:39.068 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:38 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:39.069 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:38 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6qa1rs3zIY2t5mCa09XumiPcjuvDJY/dItmJWn7Ygr0x8xUMly+ZkNj+H2PNHcYIihRlDgnrwfEDLRKOKqLYLCt6OyJ/yfQpkl5riJxx4zJCD2mJMrZsmUpbtxadaTaKQ0epsF6zy8ErxD1QzgDwCnxUdWJ1XKL0becjx65XGC0hyrnEPN6Gwn4dUAekJ+gY55jBjfiWWaYGRmgqb0Jd45OmNFRjCID+M995dKPS18USdRNFpgBcdUEOgr/CJBIdvHMJKf+NkUDPrnhvhB0Ks420z3Y9aS+qCrSaQhKPkN5pkeJd82O6OfSOHJmcHrjca464ek4SMgBJbU3Mn7Eh5b+ZntwHBCBSDX89Ity1tAG9jwu6uDtOpX9qt0veYkzFkyQs6HiayizOBIaxa885oeGoh+xsBbFIqUFf7cDLq33w1cQJSll6zR2098yzLm47pnr4L7agWitz4hm93R/H11+BhFheqrLc22Phk0/iosQa6uoOGPtRo3TKElWnXDWc= ceph-573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd66298b700 1 Processor -- start 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd66298b700 1 -- start start 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd66298b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd66298b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c102c80 con 0x7fd65c076be0 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd65bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd65bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58854/0 (socket says 172.21.15.134:58854) 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.875+0000 7fd65bfff700 1 -- 172.21.15.134:0/556239450 learned_addr learned my addr 172.21.15.134:0/556239450 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd65bfff700 1 -- 172.21.15.134:0/556239450 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c1077d0 con 0x7fd65c076be0 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd65bfff700 1 --2- 172.21.15.134:0/556239450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0098e0 tx=0x7fd64c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6fd0fc7a4460ce server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd65affd700 1 -- 172.21.15.134:0/556239450 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd64c00b860 con 0x7fd65c076be0 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd65affd700 1 -- 172.21.15.134:0/556239450 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd64c00be30 con 0x7fd65c076be0 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd65affd700 1 -- 172.21.15.134:0/556239450 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd64c010380 con 0x7fd65c076be0 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd66298b700 1 -- 172.21.15.134:0/556239450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 msgr2=0x7fd65c107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.876+0000 7fd66298b700 1 --2- 172.21.15.134:0/556239450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0098e0 tx=0x7fd64c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- 172.21.15.134:0/556239450 shutdown_connections 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 --2- 172.21.15.134:0/556239450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c107290 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- 172.21.15.134:0/556239450 >> 172.21.15.134:0/556239450 conn(0x7fd65c0fc8a0 msgr2=0x7fd65c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- 172.21.15.134:0/556239450 shutdown_connections 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- 172.21.15.134:0/556239450 wait complete. 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 Processor -- start 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- start start 2024-01-19T06:18:39.711 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.877+0000 7fd66298b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c113750 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd65bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd65bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58870/0 (socket says 172.21.15.134:58870) 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd65bfff700 1 -- 172.21.15.134:0/73852344 learned_addr learned my addr 172.21.15.134:0/73852344 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd65bfff700 1 -- 172.21.15.134:0/73852344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c0094f0 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd65bfff700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0090b0 tx=0x7fd64c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd64c010a80 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c113950 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd64c01a430 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd64c0245b0 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.878+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65c113d60 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.879+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd64c02d3f0 con 0x7fd65c076be0 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.879+0000 7fd658ff9700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 0x7fd644038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:39.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.879+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd64c04a340 con 0x7fd65c076be0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.879+0000 7fd65b7fe700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 0x7fd644038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.879+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c04bb80 con 0x7fd65c076be0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.880+0000 7fd65b7fe700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 0x7fd644038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd650009990 tx=0x7fd650006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:38.884+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd64c02b8e0 con 0x7fd65c076be0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.069+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd65c0febd0 con 0x7fd6440365f0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.069+0000 7fd658ff9700 1 -- 172.21.15.134:0/73852344 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd65c0febd0 con 0x7fd6440365f0 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 msgr2=0x7fd644038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 0x7fd644038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd650009990 tx=0x7fd650006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 msgr2=0x7fd65c113210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0090b0 tx=0x7fd64c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 shutdown_connections 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd6440365f0 0x7fd644038aa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 --2- 172.21.15.134:0/73852344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd65c076be0 0x7fd65c113210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:39.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.071+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 >> 172.21.15.134:0/73852344 conn(0x7fd65c0fc8a0 msgr2=0x7fd65c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:39.714 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.072+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 shutdown_connections 2024-01-19T06:18:39.714 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:39.072+0000 7fd66298b700 1 -- 172.21.15.134:0/73852344 wait complete. 2024-01-19T06:18:39.714 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-19T06:18:39.714 INFO:teuthology.orchestra.run.smithi134.stdout:Adding key to root@localhost authorized_keys... 2024-01-19T06:18:39.714 INFO:teuthology.orchestra.run.smithi134.stdout:Adding host smithi134... 2024-01-19T06:18:40.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:39 smithi134 ceph-mon[28283]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:41.164 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:40 smithi134 ceph-mon[28283]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "addr": "172.21.15.134", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:42.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:41 smithi134 ceph-mon[28283]: Deploying cephadm binary to smithi134 2024-01-19T06:18:42.926 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Added host 'smithi134' with addr '172.21.15.134' 2024-01-19T06:18:42.926 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f39160ef700 1 Processor -- start 2024-01-19T06:18:42.926 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f39160ef700 1 -- start start 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f39160ef700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f39160ef700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3910105420 con 0x7f3910076c20 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58884/0 (socket says 172.21.15.134:58884) 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390f7fe700 1 -- 172.21.15.134:0/3913205528 learned_addr learned my addr 172.21.15.134:0/3913205528 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390f7fe700 1 -- 172.21.15.134:0/3913205528 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910105560 con 0x7f3910076c20 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390f7fe700 1 --2- 172.21.15.134:0/3913205528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39040098e0 tx=0x7f3904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b1e1a999e6b356 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.096+0000 7f390e7fc700 1 -- 172.21.15.134:0/3913205528 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f390400b860 con 0x7f3910076c20 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f390e7fc700 1 -- 172.21.15.134:0/3913205528 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f390400be30 con 0x7f3910076c20 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f390e7fc700 1 -- 172.21.15.134:0/3913205528 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3904010380 con 0x7f3910076c20 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 -- 172.21.15.134:0/3913205528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 msgr2=0x7f3910104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 --2- 172.21.15.134:0/3913205528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f39040098e0 tx=0x7f3904009250 comp rx=0 tx=0).stop 2024-01-19T06:18:42.927 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 -- 172.21.15.134:0/3913205528 shutdown_connections 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 --2- 172.21.15.134:0/3913205528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910104ee0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 -- 172.21.15.134:0/3913205528 >> 172.21.15.134:0/3913205528 conn(0x7f39100fc950 msgr2=0x7f39100fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 -- 172.21.15.134:0/3913205528 shutdown_connections 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.097+0000 7f39160ef700 1 -- 172.21.15.134:0/3913205528 wait complete. 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f39160ef700 1 Processor -- start 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f39160ef700 1 -- start start 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f39160ef700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f39160ef700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3910100de0 con 0x7f3910076c20 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f390f7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58892/0 (socket says 172.21.15.134:58892) 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f390f7fe700 1 -- 172.21.15.134:0/4006771237 learned_addr learned my addr 172.21.15.134:0/4006771237 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f390f7fe700 1 -- 172.21.15.134:0/4006771237 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040094f0 con 0x7f3910076c20 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.098+0000 7f390f7fe700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3904006d40 tx=0x7f3904009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:42.928 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39040109b0 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3904024440 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3910100fe0 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f390401a5f0 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39101013f0 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.099+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f390401a820 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.100+0000 7f38f7fff700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 0x7f38f8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.100+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391004bb80 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.100+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f390404a550 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.100+0000 7f390effd700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 0x7f38f8038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.100+0000 7f390effd700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 0x7f38f8038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3900009990 tx=0x7f3900006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.104+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3904018a90 con 0x7f3910076c20 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:40.293+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "addr": "172.21.15.134", "target": ["mon-mgr", ""]}) v1 -- 0x7f391010adb0 con 0x7f38f8036640 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.327+0000 7f38f7fff700 1 -- 172.21.15.134:0/4006771237 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f391010adb0 con 0x7f38f8036640 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 msgr2=0x7f38f8038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:42.929 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 0x7f38f8038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3900009990 tx=0x7f3900006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 msgr2=0x7f3910102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3904006d40 tx=0x7f3904009250 comp rx=0 tx=0).stop 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 shutdown_connections 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f38f8036640 0x7f38f8038af0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 --2- 172.21.15.134:0/4006771237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3910076c20 0x7f3910102960 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 >> 172.21.15.134:0/4006771237 conn(0x7f39100fc950 msgr2=0x7f39100fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 shutdown_connections 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:42.330+0000 7f39160ef700 1 -- 172.21.15.134:0/4006771237 wait complete. 2024-01-19T06:18:42.930 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying mon service with default placement... 2024-01-19T06:18:43.567 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:43 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:43.567 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:43 smithi134 ceph-mon[28283]: Added host smithi134 2024-01-19T06:18:43.567 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:43 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:18:44.340 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-01-19T06:18:44.340 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.404+0000 7eff68463700 1 Processor -- start 2024-01-19T06:18:44.340 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.404+0000 7eff68463700 1 -- start start 2024-01-19T06:18:44.340 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.404+0000 7eff68463700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:44.340 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.404+0000 7eff68463700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60103100 con 0x7eff601027b0 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff661ff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff661ff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58906/0 (socket says 172.21.15.134:58906) 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff661ff700 1 -- 172.21.15.134:0/2422660714 learned_addr learned my addr 172.21.15.134:0/2422660714 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff661ff700 1 -- 172.21.15.134:0/2422660714 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60103240 con 0x7eff601027b0 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff661ff700 1 --2- 172.21.15.134:0/2422660714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0098e0 tx=0x7eff5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27da2fba2bb3799 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:44.341 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff651fd700 1 -- 172.21.15.134:0/2422660714 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c00b860 con 0x7eff601027b0 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff651fd700 1 -- 172.21.15.134:0/2422660714 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff5c00be30 con 0x7eff601027b0 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.405+0000 7eff651fd700 1 -- 172.21.15.134:0/2422660714 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c010380 con 0x7eff601027b0 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 -- 172.21.15.134:0/2422660714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 msgr2=0x7eff60102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 --2- 172.21.15.134:0/2422660714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0098e0 tx=0x7eff5c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 -- 172.21.15.134:0/2422660714 shutdown_connections 2024-01-19T06:18:44.342 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 --2- 172.21.15.134:0/2422660714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60102bc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:44.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 -- 172.21.15.134:0/2422660714 >> 172.21.15.134:0/2422660714 conn(0x7eff600fc0b0 msgr2=0x7eff600fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:44.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 -- 172.21.15.134:0/2422660714 shutdown_connections 2024-01-19T06:18:44.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.406+0000 7eff68463700 1 -- 172.21.15.134:0/2422660714 wait complete. 2024-01-19T06:18:44.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff68463700 1 Processor -- start 2024-01-19T06:18:44.343 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff68463700 1 -- start start 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff68463700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff68463700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60110580 con 0x7eff601027b0 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff661ff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff661ff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58916/0 (socket says 172.21.15.134:58916) 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff661ff700 1 -- 172.21.15.134:0/2227239942 learned_addr learned my addr 172.21.15.134:0/2227239942 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:44.344 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff661ff700 1 -- 172.21.15.134:0/2227239942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0094f0 con 0x7eff601027b0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff661ff700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff5c000c00 tx=0x7eff5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c010a80 con 0x7eff601027b0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff60110780 con 0x7eff601027b0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff5c01a430 con 0x7eff601027b0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.407+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5c0245b0 con 0x7eff601027b0 2024-01-19T06:18:44.345 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.408+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff60110b90 con 0x7eff601027b0 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.408+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7eff5c0247f0 con 0x7eff601027b0 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.408+0000 7eff56ffd700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 0x7eff4c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.408+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7eff5c01a5a0 con 0x7eff601027b0 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.408+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6004bb80 con 0x7eff601027b0 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.409+0000 7eff659fe700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 0x7eff4c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:44.346 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.409+0000 7eff659fe700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 0x7eff4c038aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff50009990 tx=0x7eff50006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.413+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7eff5c018350 con 0x7eff601027b0 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.620+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7eff60109e40 con 0x7eff4c0365f0 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.625+0000 7eff56ffd700 1 -- 172.21.15.134:0/2227239942 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7eff60109e40 con 0x7eff4c0365f0 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 msgr2=0x7eff4c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 0x7eff4c038aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff50009990 tx=0x7eff50006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:44.347 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 msgr2=0x7eff60110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff5c000c00 tx=0x7eff5c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 shutdown_connections 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7eff4c0365f0 0x7eff4c038aa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 --2- 172.21.15.134:0/2227239942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff601027b0 0x7eff60110040 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.627+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 >> 172.21.15.134:0/2227239942 conn(0x7eff600fc0b0 msgr2=0x7eff600fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.628+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 shutdown_connections 2024-01-19T06:18:44.349 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:43.628+0000 7eff68463700 1 -- 172.21.15.134:0/2227239942 wait complete. 2024-01-19T06:18:44.349 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying mgr service with default placement... 2024-01-19T06:18:44.880 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:44 smithi134 ceph-mon[28283]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:44.880 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:44 smithi134 ceph-mon[28283]: Saving service mon spec with placement count:5 2024-01-19T06:18:44.881 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:44 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:46.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:46 smithi134 ceph-mon[28283]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:46.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:46 smithi134 ceph-mon[28283]: Saving service mgr spec with placement count:2 2024-01-19T06:18:46.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:46 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:46.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:46 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:46.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:46 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:46.149 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.880+0000 7f227c5ec700 1 Processor -- start 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.880+0000 7f227c5ec700 1 -- start start 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227c5ec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227c5ec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2274077730 con 0x7f2274078de0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227a388700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227a388700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58932/0 (socket says 172.21.15.134:58932) 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227a388700 1 -- 172.21.15.134:0/3676801979 learned_addr learned my addr 172.21.15.134:0/3676801979 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227a388700 1 -- 172.21.15.134:0/3676801979 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2274077870 con 0x7f2274078de0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.881+0000 7f227a388700 1 --2- 172.21.15.134:0/3676801979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22700098e0 tx=0x7f2270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780fdf2653c8f807 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f2279386700 1 -- 172.21.15.134:0/3676801979 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f227000b860 con 0x7f2274078de0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f2279386700 1 -- 172.21.15.134:0/3676801979 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f227000be30 con 0x7f2274078de0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f2279386700 1 -- 172.21.15.134:0/3676801979 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2270010380 con 0x7f2274078de0 2024-01-19T06:18:46.150 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 -- 172.21.15.134:0/3676801979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 msgr2=0x7f22740771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 --2- 172.21.15.134:0/3676801979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f22700098e0 tx=0x7f2270009250 comp rx=0 tx=0).stop 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 -- 172.21.15.134:0/3676801979 shutdown_connections 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 --2- 172.21.15.134:0/3676801979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f22740771f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 -- 172.21.15.134:0/3676801979 >> 172.21.15.134:0/3676801979 conn(0x7f22740fda40 msgr2=0x7f22740ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 -- 172.21.15.134:0/3676801979 shutdown_connections 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.882+0000 7f227c5ec700 1 -- 172.21.15.134:0/3676801979 wait complete. 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227c5ec700 1 Processor -- start 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227c5ec700 1 -- start start 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227c5ec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227c5ec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2274113840 con 0x7f2274078de0 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227a388700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227a388700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58934/0 (socket says 172.21.15.134:58934) 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227a388700 1 -- 172.21.15.134:0/4224550250 learned_addr learned my addr 172.21.15.134:0/4224550250 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227a388700 1 -- 172.21.15.134:0/4224550250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22700094f0 con 0x7f2274078de0 2024-01-19T06:18:46.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.883+0000 7f227a388700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f22700090b0 tx=0x7f2270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22700109b0 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f227001a430 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22700245b0 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2274113a40 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2274113e50 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.884+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2270010b10 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.885+0000 7f226affd700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 0x7f2260038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.885+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2274108630 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.885+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f227004a630 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.885+0000 7f2279b87700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 0x7f2260038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.885+0000 7f2279b87700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 0x7f2260038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2264009990 tx=0x7f2264006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:44.889+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f227002e440 con 0x7f2274078de0 2024-01-19T06:18:46.152 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.086+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f227410d270 con 0x7f22600365f0 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.091+0000 7f226affd700 1 -- 172.21.15.134:0/4224550250 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f227410d270 con 0x7f22600365f0 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 msgr2=0x7f2260038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 0x7f2260038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2264009990 tx=0x7f2264006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 msgr2=0x7f2274113300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f22700090b0 tx=0x7f2270009250 comp rx=0 tx=0).stop 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 shutdown_connections 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f22600365f0 0x7f2260038aa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 --2- 172.21.15.134:0/4224550250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2274078de0 0x7f2274113300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.093+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 >> 172.21.15.134:0/4224550250 conn(0x7f22740fda40 msgr2=0x7f22740ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:46.153 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.094+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 shutdown_connections 2024-01-19T06:18:46.154 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:45.094+0000 7f227c5ec700 1 -- 172.21.15.134:0/4224550250 wait complete. 2024-01-19T06:18:46.154 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying crash service with default placement... 2024-01-19T06:18:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-01-19T06:18:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99ed671700 1 Processor -- start 2024-01-19T06:18:47.776 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99ed671700 1 -- start start 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99ed671700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99ed671700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e8101b20 con 0x7f99e81011d0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99e6ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99e6ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39390/0 (socket says 172.21.15.134:39390) 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99e6ffd700 1 -- 172.21.15.134:0/3447538939 learned_addr learned my addr 172.21.15.134:0/3447538939 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.517+0000 7f99e6ffd700 1 -- 172.21.15.134:0/3447538939 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8101c60 con 0x7f99e81011d0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.518+0000 7f99e6ffd700 1 --2- 172.21.15.134:0/3447538939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99d80098e0 tx=0x7f99d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a68d219a5d11acb server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.518+0000 7f99e5ffb700 1 -- 172.21.15.134:0/3447538939 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d800b860 con 0x7f99e81011d0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.518+0000 7f99e5ffb700 1 -- 172.21.15.134:0/3447538939 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99d800be70 con 0x7f99e81011d0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.518+0000 7f99e5ffb700 1 -- 172.21.15.134:0/3447538939 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d8010380 con 0x7f99e81011d0 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- 172.21.15.134:0/3447538939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 msgr2=0x7f99e81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 --2- 172.21.15.134:0/3447538939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99d80098e0 tx=0x7f99d8009250 comp rx=0 tx=0).stop 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- 172.21.15.134:0/3447538939 shutdown_connections 2024-01-19T06:18:47.777 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 --2- 172.21.15.134:0/3447538939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e81015e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- 172.21.15.134:0/3447538939 >> 172.21.15.134:0/3447538939 conn(0x7f99e80fc970 msgr2=0x7f99e80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- 172.21.15.134:0/3447538939 shutdown_connections 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- 172.21.15.134:0/3447538939 wait complete. 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 Processor -- start 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.519+0000 7f99ed671700 1 -- start start 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99ed671700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99ed671700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e81116c0 con 0x7f99e81011d0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99e6ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99e6ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39400/0 (socket says 172.21.15.134:39400) 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99e6ffd700 1 -- 172.21.15.134:0/583701868 learned_addr learned my addr 172.21.15.134:0/583701868 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99e6ffd700 1 -- 172.21.15.134:0/583701868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80094f0 con 0x7f99e81011d0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99e6ffd700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c010 tx=0x7f99d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d8010960 con 0x7f99e81011d0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99d801a430 con 0x7f99e81011d0 2024-01-19T06:18:47.778 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e81118c0 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.520+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d8019580 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.521+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e8111cd0 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.521+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f99d8019800 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.521+0000 7f99cf7fe700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 0x7f99d0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.521+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f99d804b3c0 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.522+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e804bb80 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.522+0000 7f99e67fc700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 0x7f99d0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.522+0000 7f99e67fc700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 0x7f99d0038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009990 tx=0x7f99dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.530+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f99d802bb80 con 0x7f99e81011d0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.737+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f99e810b0e0 con 0x7f99d00365f0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.741+0000 7f99cf7fe700 1 -- 172.21.15.134:0/583701868 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f99e810b0e0 con 0x7f99d00365f0 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.743+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 msgr2=0x7f99d0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.743+0000 7f99ed671700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 0x7f99d0038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009990 tx=0x7f99dc006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:47.779 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.743+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 msgr2=0x7f99e8111180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.743+0000 7f99ed671700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f99d800c010 tx=0x7f99d8009250 comp rx=0 tx=0).stop 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 shutdown_connections 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f99d00365f0 0x7f99d0038aa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 --2- 172.21.15.134:0/583701868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99e81011d0 0x7f99e8111180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 >> 172.21.15.134:0/583701868 conn(0x7f99e80fc970 msgr2=0x7f99e80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 shutdown_connections 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:46.744+0000 7f99ed671700 1 -- 172.21.15.134:0/583701868 wait complete. 2024-01-19T06:18:47.780 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying prometheus service with default placement... 2024-01-19T06:18:47.804 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:47 smithi134 ceph-mon[28283]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:47.804 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:47 smithi134 ceph-mon[28283]: Saving service crash spec with placement * 2024-01-19T06:18:47.804 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:47 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:47.805 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:47 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c1df1f700 1 Processor -- start 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c1df1f700 1 -- start start 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c1df1f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c1df1f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18101b20 con 0x7f2c181011d0 2024-01-19T06:18:48.987 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c177fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c177fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39404/0 (socket says 172.21.15.134:39404) 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c177fe700 1 -- 172.21.15.134:0/4035606161 learned_addr learned my addr 172.21.15.134:0/4035606161 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c177fe700 1 -- 172.21.15.134:0/4035606161 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18101c60 con 0x7f2c181011d0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.199+0000 7f2c177fe700 1 --2- 172.21.15.134:0/4035606161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0098e0 tx=0x7f2c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ae62042a93bd71 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c167fc700 1 -- 172.21.15.134:0/4035606161 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0c00b860 con 0x7f2c181011d0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c167fc700 1 -- 172.21.15.134:0/4035606161 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c0c00be70 con 0x7f2c181011d0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c167fc700 1 -- 172.21.15.134:0/4035606161 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0c010380 con 0x7f2c181011d0 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 -- 172.21.15.134:0/4035606161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 msgr2=0x7f2c181015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/4035606161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0098e0 tx=0x7f2c0c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 -- 172.21.15.134:0/4035606161 shutdown_connections 2024-01-19T06:18:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/4035606161 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c181015e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 -- 172.21.15.134:0/4035606161 >> 172.21.15.134:0/4035606161 conn(0x7f2c180fc970 msgr2=0x7f2c180fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 -- 172.21.15.134:0/4035606161 shutdown_connections 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.200+0000 7f2c1df1f700 1 -- 172.21.15.134:0/4035606161 wait complete. 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.201+0000 7f2c1df1f700 1 Processor -- start 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.201+0000 7f2c1df1f700 1 -- start start 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.201+0000 7f2c1df1f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.201+0000 7f2c1df1f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18070ff0 con 0x7f2c181011d0 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c177fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c177fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39406/0 (socket says 172.21.15.134:39406) 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c177fe700 1 -- 172.21.15.134:0/1003738742 learned_addr learned my addr 172.21.15.134:0/1003738742 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c177fe700 1 -- 172.21.15.134:0/1003738742 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0094f0 con 0x7f2c181011d0 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c177fe700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c011b60 tx=0x7f2c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0c010850 con 0x7f2c181011d0 2024-01-19T06:18:48.989 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c0c010e60 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c180711f0 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.202+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18114530 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c0c01ae20 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2c0c02d3f0 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2bfffff700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 0x7f2c00038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2c0c04b2e0 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2c16ffd700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 0x7f2c00038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.203+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1810a090 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.204+0000 7f2c16ffd700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 0x7f2c00038b40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009990 tx=0x7f2c08006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.210+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2c0c02b8e0 con 0x7f2c181011d0 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.414+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c1805e080 con 0x7f2c00036690 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.420+0000 7f2bfffff700 1 -- 172.21.15.134:0/1003738742 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f2c1805e080 con 0x7f2c00036690 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 msgr2=0x7f2c00038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:48.990 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 0x7f2c00038b40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009990 tx=0x7f2c08006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 msgr2=0x7f2c18113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c011b60 tx=0x7f2c0c009250 comp rx=0 tx=0).stop 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 shutdown_connections 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f2c00036690 0x7f2c00038b40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 --2- 172.21.15.134:0/1003738742 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c181011d0 0x7f2c18113e30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 >> 172.21.15.134:0/1003738742 conn(0x7f2c180fc970 msgr2=0x7f2c180fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 shutdown_connections 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:48.422+0000 7f2c1df1f700 1 -- 172.21.15.134:0/1003738742 wait complete. 2024-01-19T06:18:48.991 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying grafana service with default placement... 2024-01-19T06:18:49.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:49 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:49.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:49 smithi134 ceph-mon[28283]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:49.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:49 smithi134 ceph-mon[28283]: Saving service prometheus spec with placement count:1 2024-01-19T06:18:49.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:49 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e62c07700 1 Processor -- start 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e62c07700 1 -- start start 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e62c07700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e62c07700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c077750 con 0x7f5e5c078e00 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e609a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e609a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39422/0 (socket says 172.21.15.134:39422) 2024-01-19T06:18:50.369 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e609a3700 1 -- 172.21.15.134:0/3731403320 learned_addr learned my addr 172.21.15.134:0/3731403320 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e609a3700 1 -- 172.21.15.134:0/3731403320 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c077890 con 0x7f5e5c078e00 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.391+0000 7f5e609a3700 1 --2- 172.21.15.134:0/3731403320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e500098e0 tx=0x7f5e50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6226ef50cbb90bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e5b7fe700 1 -- 172.21.15.134:0/3731403320 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5000b860 con 0x7f5e5c078e00 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e5b7fe700 1 -- 172.21.15.134:0/3731403320 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e5000be70 con 0x7f5e5c078e00 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e5b7fe700 1 -- 172.21.15.134:0/3731403320 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e50010380 con 0x7f5e5c078e00 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 -- 172.21.15.134:0/3731403320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 msgr2=0x7f5e5c077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3731403320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e500098e0 tx=0x7f5e50009250 comp rx=0 tx=0).stop 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 -- 172.21.15.134:0/3731403320 shutdown_connections 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3731403320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c077210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 -- 172.21.15.134:0/3731403320 >> 172.21.15.134:0/3731403320 conn(0x7f5e5c0fda40 msgr2=0x7f5e5c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 -- 172.21.15.134:0/3731403320 shutdown_connections 2024-01-19T06:18:50.370 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.392+0000 7f5e62c07700 1 -- 172.21.15.134:0/3731403320 wait complete. 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e62c07700 1 Processor -- start 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e62c07700 1 -- start start 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e62c07700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e62c07700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c113890 con 0x7f5e5c078e00 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e609a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e609a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39436/0 (socket says 172.21.15.134:39436) 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e609a3700 1 -- 172.21.15.134:0/3299733458 learned_addr learned my addr 172.21.15.134:0/3299733458 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.393+0000 7f5e609a3700 1 -- 172.21.15.134:0/3299733458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e500094f0 con 0x7f5e5c078e00 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e609a3700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000ba70 tx=0x7f5e50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e50010960 con 0x7f5e5c078e00 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e50024430 con 0x7f5e5c078e00 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c113a90 con 0x7f5e5c078e00 2024-01-19T06:18:50.371 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5001a5f0 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c113ea0 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.394+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5e5001a860 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.395+0000 7f5e597fa700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 0x7f5e4c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.395+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5e500249f0 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.395+0000 7f5e5bfff700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 0x7f5e4c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.395+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c05f270 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.395+0000 7f5e5bfff700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 0x7f5e4c038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009990 tx=0x7f5e48006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.400+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e50018db0 con 0x7f5e5c078e00 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.603+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e5c10ceb0 con 0x7f5e4c036640 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.608+0000 7f5e597fa700 1 -- 172.21.15.134:0/3299733458 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5e5c10ceb0 con 0x7f5e4c036640 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 msgr2=0x7f5e4c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:50.372 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 0x7f5e4c038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009990 tx=0x7f5e48006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 msgr2=0x7f5e5c113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000ba70 tx=0x7f5e50009250 comp rx=0 tx=0).stop 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 shutdown_connections 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f5e4c036640 0x7f5e4c038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 --2- 172.21.15.134:0/3299733458 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e5c078e00 0x7f5e5c113350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 >> 172.21.15.134:0/3299733458 conn(0x7f5e5c0fda40 msgr2=0x7f5e5c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 shutdown_connections 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:49.610+0000 7f5e62c07700 1 -- 172.21.15.134:0/3299733458 wait complete. 2024-01-19T06:18:50.373 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying node-exporter service with default placement... 2024-01-19T06:18:50.638 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:50 smithi134 ceph-mon[28283]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:50.639 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:50 smithi134 ceph-mon[28283]: Saving service grafana spec with placement count:1 2024-01-19T06:18:50.639 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:50 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:52.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-01-19T06:18:52.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78dcbb4700 1 Processor -- start 2024-01-19T06:18:52.187 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78dcbb4700 1 -- start start 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78dcbb4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78dcbb4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8101b20 con 0x7f78d81011d0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78d659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78d659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39446/0 (socket says 172.21.15.134:39446) 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.329+0000 7f78d659c700 1 -- 172.21.15.134:0/732277271 learned_addr learned my addr 172.21.15.134:0/732277271 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78d659c700 1 -- 172.21.15.134:0/732277271 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8101c60 con 0x7f78d81011d0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78d659c700 1 --2- 172.21.15.134:0/732277271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c800bb80 tx=0x7f78c800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef740ffa1bbd5405 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78d559a700 1 -- 172.21.15.134:0/732277271 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c800dd00 con 0x7f78d81011d0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78d559a700 1 -- 172.21.15.134:0/732277271 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78c800de60 con 0x7f78d81011d0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78d559a700 1 -- 172.21.15.134:0/732277271 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c80188d0 con 0x7f78d81011d0 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78dcbb4700 1 -- 172.21.15.134:0/732277271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 msgr2=0x7f78d81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.330+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/732277271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78c800bb80 tx=0x7f78c800b4f0 comp rx=0 tx=0).stop 2024-01-19T06:18:52.188 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- 172.21.15.134:0/732277271 shutdown_connections 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/732277271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d81015e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- 172.21.15.134:0/732277271 >> 172.21.15.134:0/732277271 conn(0x7f78d80fc950 msgr2=0x7f78d80fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- 172.21.15.134:0/732277271 shutdown_connections 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- 172.21.15.134:0/732277271 wait complete. 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 Processor -- start 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- start start 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78dcbb4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d810f440 con 0x7f78d81011d0 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78d659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78d659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39452/0 (socket says 172.21.15.134:39452) 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.331+0000 7f78d659c700 1 -- 172.21.15.134:0/1051422521 learned_addr learned my addr 172.21.15.134:0/1051422521 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78d659c700 1 -- 172.21.15.134:0/1051422521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c800b790 con 0x7f78d81011d0 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78d659c700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78c80184f0 tx=0x7f78c800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c8010bd0 con 0x7f78d81011d0 2024-01-19T06:18:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78c801d050 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d810f640 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c8022590 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.332+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d810fa50 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f78c8022800 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78c6ffd700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 0x7f78c0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f78c8014070 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78d5d9b700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 0x7f78c0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d804bb80 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.333+0000 7f78d5d9b700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 0x7f78c0038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009990 tx=0x7f78cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.338+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f78c8022b00 con 0x7f78d81011d0 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.531+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f78d8108950 con 0x7f78c0036640 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.536+0000 7f78c6ffd700 1 -- 172.21.15.134:0/1051422521 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f78d8108950 con 0x7f78c0036640 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 msgr2=0x7f78c0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:52.190 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 0x7f78c0038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009990 tx=0x7f78cc006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 msgr2=0x7f78d810ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f78c80184f0 tx=0x7f78c800b4f0 comp rx=0 tx=0).stop 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 shutdown_connections 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f78c0036640 0x7f78c0038af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 --2- 172.21.15.134:0/1051422521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d81011d0 0x7f78d810ef00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 >> 172.21.15.134:0/1051422521 conn(0x7f78d80fc950 msgr2=0x7f78d8105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 shutdown_connections 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:51.538+0000 7f78dcbb4700 1 -- 172.21.15.134:0/1051422521 wait complete. 2024-01-19T06:18:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying alertmanager service with default placement... 2024-01-19T06:18:52.760 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:52 smithi134 ceph-mon[28283]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:52.760 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:52 smithi134 ceph-mon[28283]: Saving service node-exporter spec with placement * 2024-01-19T06:18:52.760 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:52 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:53.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-01-19T06:18:53.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7d5405700 1 Processor -- start 2024-01-19T06:18:53.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7d5405700 1 -- start start 2024-01-19T06:18:53.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7d5405700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7d5405700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01042e0 con 0x7fb7d0103990 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39468/0 (socket says 172.21.15.134:39468) 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7ceffd700 1 -- 172.21.15.134:0/896572495 learned_addr learned my addr 172.21.15.134:0/896572495 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.560+0000 7fb7ceffd700 1 -- 172.21.15.134:0/896572495 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d0104420 con 0x7fb7d0103990 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7ceffd700 1 --2- 172.21.15.134:0/896572495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00098e0 tx=0x7fb7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d799a807f372d00 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7cdffb700 1 -- 172.21.15.134:0/896572495 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c000b860 con 0x7fb7d0103990 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7cdffb700 1 -- 172.21.15.134:0/896572495 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7c000be70 con 0x7fb7d0103990 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7cdffb700 1 -- 172.21.15.134:0/896572495 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c0010380 con 0x7fb7d0103990 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 -- 172.21.15.134:0/896572495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 msgr2=0x7fb7d0103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 --2- 172.21.15.134:0/896572495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00098e0 tx=0x7fb7c0009250 comp rx=0 tx=0).stop 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 -- 172.21.15.134:0/896572495 shutdown_connections 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 --2- 172.21.15.134:0/896572495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d0103da0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:53.303 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 -- 172.21.15.134:0/896572495 >> 172.21.15.134:0/896572495 conn(0x7fb7d00fc0b0 msgr2=0x7fb7d00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 -- 172.21.15.134:0/896572495 shutdown_connections 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.561+0000 7fb7d5405700 1 -- 172.21.15.134:0/896572495 wait complete. 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7d5405700 1 Processor -- start 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7d5405700 1 -- start start 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7d5405700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7d5405700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d010f4a0 con 0x7fb7d0103990 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39476/0 (socket says 172.21.15.134:39476) 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.562+0000 7fb7ceffd700 1 -- 172.21.15.134:0/1383025155 learned_addr learned my addr 172.21.15.134:0/1383025155 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7ceffd700 1 -- 172.21.15.134:0/1383025155 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00094f0 con 0x7fb7d0103990 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7ceffd700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c010 tx=0x7fb7c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c0010960 con 0x7fb7d0103990 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7c001a430 con 0x7fb7d0103990 2024-01-19T06:18:53.304 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7c0019580 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d010f6a0 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.563+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d010fab0 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb7c00197b0 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d004bb80 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7b77fe700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 0x7fb7b803cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7c004a470 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7ce7fc700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 0x7fb7b803cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.564+0000 7fb7ce7fc700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 0x7fb7b803cf40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009990 tx=0x7fb7c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.569+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb7c002bd60 con 0x7fb7d0103990 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.761+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d00fd5a0 con 0x7fb7b803aa90 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.765+0000 7fb7b77fe700 1 -- 172.21.15.134:0/1383025155 <== mgr.14118 v2:172.21.15.134:6800/4139544225 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb7d00fd5a0 con 0x7fb7b803aa90 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 msgr2=0x7fb7b803cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 0x7fb7b803cf40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4009990 tx=0x7fb7c4006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:53.305 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 msgr2=0x7fb7d010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c010 tx=0x7fb7c0009250 comp rx=0 tx=0).stop 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 shutdown_connections 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fb7b803aa90 0x7fb7b803cf40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 --2- 172.21.15.134:0/1383025155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7d0103990 0x7fb7d010ef60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 >> 172.21.15.134:0/1383025155 conn(0x7fb7d00fc0b0 msgr2=0x7fb7d00fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 shutdown_connections 2024-01-19T06:18:53.306 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:52.767+0000 7fb7d5405700 1 -- 172.21.15.134:0/1383025155 wait complete. 2024-01-19T06:18:53.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:53 smithi134 ceph-mon[28283]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:18:53.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:53 smithi134 ceph-mon[28283]: Saving service alertmanager spec with placement count:1 2024-01-19T06:18:53.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:53 smithi134 ceph-mon[28283]: from='mgr.14118 172.21.15.134:0/3372080699' entity='mgr.smithi134.qrxokr' 2024-01-19T06:18:53.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:53 smithi134 ceph-mon[28283]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:18:54.442 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f73a2c700 1 Processor -- start 2024-01-19T06:18:54.442 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f73a2c700 1 -- start start 2024-01-19T06:18:54.442 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f73a2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f73a2c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c102c00 con 0x7f6f6c1022b0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f717c8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f717c8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39480/0 (socket says 172.21.15.134:39480) 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.670+0000 7f6f717c8700 1 -- 172.21.15.134:0/3818149795 learned_addr learned my addr 172.21.15.134:0/3818149795 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f717c8700 1 -- 172.21.15.134:0/3818149795 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c102d40 con 0x7f6f6c1022b0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f717c8700 1 --2- 172.21.15.134:0/3818149795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f680098e0 tx=0x7f6f68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4395e08abb3b1375 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f63fff700 1 -- 172.21.15.134:0/3818149795 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f6800b860 con 0x7f6f6c1022b0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f63fff700 1 -- 172.21.15.134:0/3818149795 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f6800be70 con 0x7f6f6c1022b0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f63fff700 1 -- 172.21.15.134:0/3818149795 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f68010380 con 0x7f6f6c1022b0 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f73a2c700 1 -- 172.21.15.134:0/3818149795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 msgr2=0x7f6f6c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.671+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/3818149795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f680098e0 tx=0x7f6f68009250 comp rx=0 tx=0).stop 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- 172.21.15.134:0/3818149795 shutdown_connections 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/3818149795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c1026c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:54.443 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- 172.21.15.134:0/3818149795 >> 172.21.15.134:0/3818149795 conn(0x7f6f6c0fda10 msgr2=0x7f6f6c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- 172.21.15.134:0/3818149795 shutdown_connections 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- 172.21.15.134:0/3818149795 wait complete. 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 Processor -- start 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- start start 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f73a2c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c111690 con 0x7f6f6c1022b0 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f717c8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f717c8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39492/0 (socket says 172.21.15.134:39492) 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.672+0000 7f6f717c8700 1 -- 172.21.15.134:0/1660131780 learned_addr learned my addr 172.21.15.134:0/1660131780 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f717c8700 1 -- 172.21.15.134:0/1660131780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f680094f0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f717c8700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800ba70 tx=0x7f6f68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f680109b0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.444 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c111890 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f6801a430 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f68024590 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.673+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c111ca0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.674+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f6f6802d3f0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.674+0000 7f6f61ffb700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 0x7f6f58038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.674+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6f6804b1a0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.674+0000 7f6f70fc7700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 0x7f6f58038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.674+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c04bb80 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.675+0000 7f6f70fc7700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 0x7f6f58038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009990 tx=0x7f6f5c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.679+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6f6802b8e0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.862+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f6f6c112630 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.868+0000 7f6f61ffb700 1 -- 172.21.15.134:0/1660131780 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6f6802b2f0 con 0x7f6f6c1022b0 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 msgr2=0x7f6f58038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:54.445 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 0x7f6f58038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c009990 tx=0x7f6f5c006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 msgr2=0x7f6f6c111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800ba70 tx=0x7f6f68009250 comp rx=0 tx=0).stop 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 shutdown_connections 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f6f580365f0 0x7f6f58038aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 --2- 172.21.15.134:0/1660131780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6f6c1022b0 0x7f6f6c111150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 >> 172.21.15.134:0/1660131780 conn(0x7f6f6c0fda10 msgr2=0x7f6f6c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 shutdown_connections 2024-01-19T06:18:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:53.870+0000 7f6f73a2c700 1 -- 172.21.15.134:0/1660131780 wait complete. 2024-01-19T06:18:55.029 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:54 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1660131780' entity='client.admin' 2024-01-19T06:18:55.561 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5ee474700 1 Processor -- start 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5ee474700 1 -- start start 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5ee474700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5ee474700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e8102c00 con 0x7fd5e81022b0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5e7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5e7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39494/0 (socket says 172.21.15.134:39494) 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.830+0000 7fd5e7fff700 1 -- 172.21.15.134:0/3391106305 learned_addr learned my addr 172.21.15.134:0/3391106305 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.831+0000 7fd5e7fff700 1 -- 172.21.15.134:0/3391106305 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e8102d40 con 0x7fd5e81022b0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.831+0000 7fd5e7fff700 1 --2- 172.21.15.134:0/3391106305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0098e0 tx=0x7fd5dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd853909f4b56a7b server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.831+0000 7fd5e6ffd700 1 -- 172.21.15.134:0/3391106305 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc00b860 con 0x7fd5e81022b0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.831+0000 7fd5e6ffd700 1 -- 172.21.15.134:0/3391106305 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5dc00be70 con 0x7fd5e81022b0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.831+0000 7fd5e6ffd700 1 -- 172.21.15.134:0/3391106305 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc010380 con 0x7fd5e81022b0 2024-01-19T06:18:55.567 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- 172.21.15.134:0/3391106305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 msgr2=0x7fd5e81026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3391106305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0098e0 tx=0x7fd5dc009250 comp rx=0 tx=0).stop 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- 172.21.15.134:0/3391106305 shutdown_connections 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3391106305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e81026c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- 172.21.15.134:0/3391106305 >> 172.21.15.134:0/3391106305 conn(0x7fd5e80fda10 msgr2=0x7fd5e80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- 172.21.15.134:0/3391106305 shutdown_connections 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- 172.21.15.134:0/3391106305 wait complete. 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 Processor -- start 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.832+0000 7fd5ee474700 1 -- start start 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5ee474700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5ee474700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e8077190 con 0x7fd5e81022b0 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39502/0 (socket says 172.21.15.134:39502) 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e7fff700 1 -- 172.21.15.134:0/3599017192 learned_addr learned my addr 172.21.15.134:0/3599017192 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:55.568 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e7fff700 1 -- 172.21.15.134:0/3599017192 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc0094f0 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e7fff700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c010 tx=0x7fd5dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc010960 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5dc01a430 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc019580 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e8077390 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.833+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e80777a0 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.834+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e804bb80 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.835+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd5dc01a980 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.835+0000 7fd5e4ff9700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 0x7fd5d0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.835+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd5dc04a4d0 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.835+0000 7fd5e77fe700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 0x7fd5d0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.836+0000 7fd5e77fe700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 0x7fd5d0038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009990 tx=0x7fd5d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:54.839+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd5dc018d20 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.030+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd5e8104090 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.033+0000 7fd5e4ff9700 1 -- 172.21.15.134:0/3599017192 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd5dc018b40 con 0x7fd5e81022b0 2024-01-19T06:18:55.569 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.035+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 msgr2=0x7fd5d0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.035+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 0x7fd5d0038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009990 tx=0x7fd5d8006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.035+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 msgr2=0x7fd5e8078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.035+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00c010 tx=0x7fd5dc009250 comp rx=0 tx=0).stop 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 shutdown_connections 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7fd5d00365f0 0x7fd5d0038aa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 --2- 172.21.15.134:0/3599017192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5e81022b0 0x7fd5e8078d10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 >> 172.21.15.134:0/3599017192 conn(0x7fd5e80fda10 msgr2=0x7fd5e80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 shutdown_connections 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.036+0000 7fd5ee474700 1 -- 172.21.15.134:0/3599017192 wait complete. 2024-01-19T06:18:55.570 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling the dashboard module... 2024-01-19T06:18:56.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:56 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3599017192' entity='client.admin' 2024-01-19T06:18:56.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:56 smithi134 ceph-mon[28283]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:18:57.382 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:57 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1319543428' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-19T06:18:57.640 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.936+0000 7f3a1fb88700 1 Processor -- start 2024-01-19T06:18:57.640 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.936+0000 7f3a1fb88700 1 -- start start 2024-01-19T06:18:57.640 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.936+0000 7f3a1fb88700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.936+0000 7f3a1fb88700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18104310 con 0x7f3a181039c0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1d924700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1d924700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39504/0 (socket says 172.21.15.134:39504) 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1d924700 1 -- 172.21.15.134:0/1939390948 learned_addr learned my addr 172.21.15.134:0/1939390948 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1d924700 1 -- 172.21.15.134:0/1939390948 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a18101fb0 con 0x7f3a181039c0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1d924700 1 --2- 172.21.15.134:0/1939390948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009e50 tx=0x7f3a14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582bb3afd99975c server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1c922700 1 -- 172.21.15.134:0/1939390948 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a140092e0 con 0x7f3a181039c0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1c922700 1 -- 172.21.15.134:0/1939390948 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a14012430 con 0x7f3a181039c0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.937+0000 7f3a1c922700 1 -- 172.21.15.134:0/1939390948 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a140075c0 con 0x7f3a181039c0 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1939390948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 msgr2=0x7f3a18103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1939390948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009e50 tx=0x7f3a14009840 comp rx=0 tx=0).stop 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1939390948 shutdown_connections 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1939390948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18103dd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:57.641 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1939390948 >> 172.21.15.134:0/1939390948 conn(0x7f3a180fda40 msgr2=0x7f3a180ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1939390948 shutdown_connections 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1939390948 wait complete. 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.938+0000 7f3a1fb88700 1 Processor -- start 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1fb88700 1 -- start start 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1fb88700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1fb88700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a181116a0 con 0x7f3a181039c0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1d924700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1d924700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39508/0 (socket says 172.21.15.134:39508) 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1d924700 1 -- 172.21.15.134:0/1319543428 learned_addr learned my addr 172.21.15.134:0/1319543428 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1d924700 1 -- 172.21.15.134:0/1319543428 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14009ae0 con 0x7f3a181039c0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1d924700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400bc80 tx=0x7f3a14009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a140092e0 con 0x7f3a181039c0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a14012900 con 0x7f3a181039c0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a181118a0 con 0x7f3a181039c0 2024-01-19T06:18:57.642 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a14025970 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.939+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a18111cb0 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.940+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f3a14019050 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.940+0000 7f3a0e7fc700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 0x7f3a04038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.940+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3a1404a060 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.940+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a18102d30 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.941+0000 7f3a1d123700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 0x7f3a04038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.941+0000 7f3a1d123700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 0x7f3a04038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009990 tx=0x7f3a08006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:55.945+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3a14020080 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:56.183+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3a1810af70 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.043+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f3a1402bc70 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.045+0000 7f3a0e7fc700 1 -- 172.21.15.134:0/1319543428 <== mon.0 v2:172.21.15.134:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3a14011560 con 0x7f3a181039c0 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 msgr2=0x7f3a04038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:57.643 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 0x7f3a04038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009990 tx=0x7f3a08006e40 comp rx=0 tx=0).stop 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 msgr2=0x7f3a18111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400bc80 tx=0x7f3a14009840 comp rx=0 tx=0).stop 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 shutdown_connections 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f3a04036640 0x7f3a04038af0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 --2- 172.21.15.134:0/1319543428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a181039c0 0x7f3a18111160 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 >> 172.21.15.134:0/1319543428 conn(0x7f3a180fda40 msgr2=0x7f3a180ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 shutdown_connections 2024-01-19T06:18:57.644 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:57.048+0000 7f3a1fb88700 1 -- 172.21.15.134:0/1319543428 wait complete. 2024-01-19T06:18:58.271 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:58 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1319543428' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-19T06:18:58.271 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:58 smithi134 ceph-mon[28283]: mgrmap e9: smithi134.qrxokr(active, since 23s) 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "smithi134.qrxokr", 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:18:58.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.066+0000 7f9702e38700 1 Processor -- start 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.066+0000 7f9702e38700 1 -- start start 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9702e38700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9702e38700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc103120 con 0x7f96fc1027d0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9700bd4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9700bd4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47174/0 (socket says 172.21.15.134:47174) 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9700bd4700 1 -- 172.21.15.134:0/3103738709 learned_addr learned my addr 172.21.15.134:0/3103738709 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9700bd4700 1 -- 172.21.15.134:0/3103738709 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc103260 con 0x7f96fc1027d0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.067+0000 7f9700bd4700 1 --2- 172.21.15.134:0/3103738709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f96f000bb80 tx=0x7f96f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f73e8248775008 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f96fb7fe700 1 -- 172.21.15.134:0/3103738709 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f000dd00 con 0x7f96fc1027d0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f96fb7fe700 1 -- 172.21.15.134:0/3103738709 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f000de60 con 0x7f96fc1027d0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f96fb7fe700 1 -- 172.21.15.134:0/3103738709 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f00188d0 con 0x7f96fc1027d0 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 -- 172.21.15.134:0/3103738709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 msgr2=0x7f96fc102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 --2- 172.21.15.134:0/3103738709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f96f000bb80 tx=0x7f96f000b4f0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.857 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 -- 172.21.15.134:0/3103738709 shutdown_connections 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 --2- 172.21.15.134:0/3103738709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc102be0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 -- 172.21.15.134:0/3103738709 >> 172.21.15.134:0/3103738709 conn(0x7f96fc0fc0b0 msgr2=0x7f96fc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 -- 172.21.15.134:0/3103738709 shutdown_connections 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.068+0000 7f9702e38700 1 -- 172.21.15.134:0/3103738709 wait complete. 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9702e38700 1 Processor -- start 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9702e38700 1 -- start start 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9702e38700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9702e38700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc10f450 con 0x7f96fc1027d0 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9700bd4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9700bd4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47190/0 (socket says 172.21.15.134:47190) 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.069+0000 7f9700bd4700 1 -- 172.21.15.134:0/3628194119 learned_addr learned my addr 172.21.15.134:0/3628194119 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f9700bd4700 1 -- 172.21.15.134:0/3628194119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f000b790 con 0x7f96fc1027d0 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f9700bd4700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96f00184f0 tx=0x7f96f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f0010bd0 con 0x7f96fc1027d0 2024-01-19T06:18:58.858 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc10f650 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f001d050 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f0022590 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.070+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc10fa60 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f96f0022850 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc04bb80 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f96f97fa700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 0x7f96ec038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f96fbfff700 1 -- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 msgr2=0x7f96ec038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f96f0014070 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.071+0000 7f96fbfff700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 0x7f96ec038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.076+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f96f000e360 con 0x7f96fc1027d0 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.272+0000 7f96fbfff700 1 -- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 msgr2=0x7f96ec038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.272+0000 7f96fbfff700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 0x7f96ec038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:18:58.859 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.321+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f96fc108570 con 0x7f96fc1027d0 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.322+0000 7f96f97fa700 1 -- 172.21.15.134:0/3628194119 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f96f0049070 con 0x7f96fc1027d0 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 msgr2=0x7f96ec038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 0x7f96ec038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 msgr2=0x7f96fc10ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96f00184f0 tx=0x7f96f000b4f0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 shutdown_connections 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f96ec036640 0x7f96ec038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 --2- 172.21.15.134:0/3628194119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96fc1027d0 0x7f96fc10ef10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 >> 172.21.15.134:0/3628194119 conn(0x7f96fc0fc0b0 msgr2=0x7f96fc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 shutdown_connections 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:58.324+0000 7f9702e38700 1 -- 172.21.15.134:0/3628194119 wait complete. 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-01-19T06:18:58.860 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 9... 2024-01-19T06:18:59.141 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:18:59 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3628194119' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: Active manager daemon smithi134.qrxokr restarted 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: Activating manager daemon smithi134.qrxokr 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: osdmap e3: 0 total, 0 up, 0 in 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: mgrmap e10: smithi134.qrxokr(active, starting, since 0.00322762s) 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.qrxokr", "id": "smithi134.qrxokr"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: Manager daemon smithi134.qrxokr is now available 2024-01-19T06:19:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:19:03.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:03 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:19:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:04 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/mirror_snapshot_schedule"}]: dispatch 2024-01-19T06:19:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:04 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/trash_purge_schedule"}]: dispatch 2024-01-19T06:19:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:04 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:04 smithi134 ceph-mon[28283]: mgrmap e11: smithi134.qrxokr(active, since 1.00574s) 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.356+0000 7f62c0f16700 1 Processor -- start 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.356+0000 7f62c0f16700 1 -- start start 2024-01-19T06:19:05.334 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62c0f16700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62c0f16700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc102c90 con 0x7f62bc071000 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62ba59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62ba59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47196/0 (socket says 172.21.15.134:47196) 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62ba59c700 1 -- 172.21.15.134:0/1869939601 learned_addr learned my addr 172.21.15.134:0/1869939601 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62ba59c700 1 -- 172.21.15.134:0/1869939601 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc1077e0 con 0x7f62bc071000 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62ba59c700 1 --2- 172.21.15.134:0/1869939601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098e0 tx=0x7f62ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b7604373460d91 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62b959a700 1 -- 172.21.15.134:0/1869939601 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac00b860 con 0x7f62bc071000 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62b959a700 1 -- 172.21.15.134:0/1869939601 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62ac00be70 con 0x7f62bc071000 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.357+0000 7f62b959a700 1 -- 172.21.15.134:0/1869939601 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac010380 con 0x7f62bc071000 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 -- 172.21.15.134:0/1869939601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 msgr2=0x7f62bc1072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 --2- 172.21.15.134:0/1869939601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098e0 tx=0x7f62ac009250 comp rx=0 tx=0).stop 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 -- 172.21.15.134:0/1869939601 shutdown_connections 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 --2- 172.21.15.134:0/1869939601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1072a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.335 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 -- 172.21.15.134:0/1869939601 >> 172.21.15.134:0/1869939601 conn(0x7f62bc0fc970 msgr2=0x7f62bc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 -- 172.21.15.134:0/1869939601 shutdown_connections 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.358+0000 7f62c0f16700 1 -- 172.21.15.134:0/1869939601 wait complete. 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62c0f16700 1 Processor -- start 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62c0f16700 1 -- start start 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62c0f16700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62c0f16700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc1138e0 con 0x7f62bc071000 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62ba59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62ba59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47204/0 (socket says 172.21.15.134:47204) 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62ba59c700 1 -- 172.21.15.134:0/783352744 learned_addr learned my addr 172.21.15.134:0/783352744 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62ba59c700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0094f0 con 0x7f62bc071000 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62ba59c700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0094c0 tx=0x7f62ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac010850 con 0x7f62bc071000 2024-01-19T06:19:05.336 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.359+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62ac010e60 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.360+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc113ae0 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.360+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac01a430 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.360+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc113ef0 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.360+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f62ac01a6b0 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.361+0000 7f62aaffd700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.361+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62a4039200 con 0x7f62a4036640 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.361+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f62ac04a620 con 0x7f62bc071000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.361+0000 7f62b9d9b700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.361+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.562+0000 7f62b9d9b700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.562+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.963+0000 7f62b9d9b700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:18:59.963+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:00.764+0000 7f62b9d9b700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:00.764+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-19T06:19:05.337 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:02.365+0000 7f62b9d9b700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/4139544225 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:02.365+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:03.685+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7f62ac0247b0 con 0x7f62bc071000 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:03.685+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 msgr2=0x7f62a4038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:03.685+0000 7f62aaffd700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.686+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f62ac018350 con 0x7f62bc071000 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.686+0000 7f62aaffd700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 0x7f62a403c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.686+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62a4039200 con 0x7f62a4039fb0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.687+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 0x7f62a403c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.688+0000 7f62b9d9b700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 0x7f62a403c390 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f62b00039d0 tx=0x7f62b00092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.689+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f62a4039200 con 0x7f62a4039fb0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f62bc04bb80 con 0x7f62a4039fb0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62aaffd700 1 -- 172.21.15.134:0/783352744 <== mgr.14160 v2:172.21.15.134:6800/1934811542 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f62bc04bb80 con 0x7f62a4039fb0 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 msgr2=0x7f62a403c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:05.338 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62c0f16700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 0x7f62a403c390 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f62b00039d0 tx=0x7f62b00092f0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 msgr2=0x7f62bc1133a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.695+0000 7f62c0f16700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0094c0 tx=0x7f62ac009250 comp rx=0 tx=0).stop 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 shutdown_connections 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f62a4039fb0 0x7f62a403c390 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62bc071000 0x7f62bc1133a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 --2- 172.21.15.134:0/783352744 >> [v2:172.21.15.134:6800/4139544225,v1:172.21.15.134:6801/4139544225] conn(0x7f62a4036640 0x7f62a4038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 >> 172.21.15.134:0/783352744 conn(0x7f62bc0fc970 msgr2=0x7f62bc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 shutdown_connections 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:04.696+0000 7f62c0f16700 1 -- 172.21.15.134:0/783352744 wait complete. 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 9 is available 2024-01-19T06:19:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:Generating a dashboard self-signed certificate... 2024-01-19T06:19:06.195 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: [19/Jan/2024:06:19:04] ENGINE Bus STARTING 2024-01-19T06:19:06.195 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: [19/Jan/2024:06:19:04] ENGINE Serving on https://172.21.15.134:7150 2024-01-19T06:19:06.195 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: [19/Jan/2024:06:19:04] ENGINE Bus STARTED 2024-01-19T06:19:06.196 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-19T06:19:06.196 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-19T06:19:06.196 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:06.196 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:05 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.727+0000 7f52e7008700 1 Processor -- start 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.727+0000 7f52e7008700 1 -- start start 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.727+0000 7f52e7008700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.727+0000 7f52e7008700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e01051d0 con 0x7f52e01047f0 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52e4da4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:06.808 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52e4da4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47266/0 (socket says 172.21.15.134:47266) 2024-01-19T06:19:06.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52e4da4700 1 -- 172.21.15.134:0/1670697796 learned_addr learned my addr 172.21.15.134:0/1670697796 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:06.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52e4da4700 1 -- 172.21.15.134:0/1670697796 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0105310 con 0x7f52e01047f0 2024-01-19T06:19:06.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52e4da4700 1 --2- 172.21.15.134:0/1670697796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52d40098e0 tx=0x7f52d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71cee69f45d40597 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:06.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52df7fe700 1 -- 172.21.15.134:0/1670697796 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d400b860 con 0x7f52e01047f0 2024-01-19T06:19:06.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52df7fe700 1 -- 172.21.15.134:0/1670697796 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52d400be70 con 0x7f52e01047f0 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.728+0000 7f52df7fe700 1 -- 172.21.15.134:0/1670697796 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d4010380 con 0x7f52e01047f0 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 -- 172.21.15.134:0/1670697796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 msgr2=0x7f52e0104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 --2- 172.21.15.134:0/1670697796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52d40098e0 tx=0x7f52d4009250 comp rx=0 tx=0).stop 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 -- 172.21.15.134:0/1670697796 shutdown_connections 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 --2- 172.21.15.134:0/1670697796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0104c00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 -- 172.21.15.134:0/1670697796 >> 172.21.15.134:0/1670697796 conn(0x7f52e0077ab0 msgr2=0x7f52e0077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 -- 172.21.15.134:0/1670697796 shutdown_connections 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.729+0000 7f52e7008700 1 -- 172.21.15.134:0/1670697796 wait complete. 2024-01-19T06:19:06.810 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.730+0000 7f52e7008700 1 Processor -- start 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.730+0000 7f52e7008700 1 -- start start 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.730+0000 7f52e7008700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.730+0000 7f52e7008700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e01138d0 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e4da4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e4da4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47282/0 (socket says 172.21.15.134:47282) 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e4da4700 1 -- 172.21.15.134:0/1843909752 learned_addr learned my addr 172.21.15.134:0/1843909752 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e4da4700 1 -- 172.21.15.134:0/1843909752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d40094f0 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e4da4700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c010 tx=0x7f52d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d4010960 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52d401a430 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e0113ad0 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52d4019580 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.731+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e0113ee0 con 0x7f52e01047f0 2024-01-19T06:19:06.811 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.732+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f52d40197b0 con 0x7f52e01047f0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.732+0000 7f52dd7fa700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 0x7f52d0038a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.732+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e004bb80 con 0x7f52e01047f0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.732+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f52d401a980 con 0x7f52e01047f0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.732+0000 7f52dffff700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 0x7f52d0038a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.733+0000 7f52dffff700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 0x7f52d0038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009990 tx=0x7f52cc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.737+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f52d402bd60 con 0x7f52e01047f0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.956+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f52d4019d30 con 0x7f52e01047f0 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:05.962+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f52e00008d0 con 0x7f52d0036570 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.230+0000 7f52dd7fa700 1 -- 172.21.15.134:0/1843909752 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f52e00008d0 con 0x7f52d0036570 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.232+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 msgr2=0x7f52d0038a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.232+0000 7f52e7008700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 0x7f52d0038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009990 tx=0x7f52cc006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.232+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 msgr2=0x7f52e0113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.232+0000 7f52e7008700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c010 tx=0x7f52d4009250 comp rx=0 tx=0).stop 2024-01-19T06:19:06.812 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 shutdown_connections 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52d0036570 0x7f52d0038a20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 --2- 172.21.15.134:0/1843909752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52e01047f0 0x7f52e0113390 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 >> 172.21.15.134:0/1843909752 conn(0x7f52e0077ab0 msgr2=0x7f52e0103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 shutdown_connections 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:06.233+0000 7f52e7008700 1 -- 172.21.15.134:0/1843909752 wait complete. 2024-01-19T06:19:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial admin user... 2024-01-19T06:19:07.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:06 smithi134 ceph-mon[28283]: mgrmap e12: smithi134.qrxokr(active, since 2s) 2024-01-19T06:19:07.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:06 smithi134 ceph-mon[28283]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:19:07.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:06 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:07.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:06 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$YBMlVdMtE6fywa.j/PJmbuT6iyDoAje.dDzVZz6v5kwAaUH.dj/ji", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705645148, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.007+0000 7fb9c2323700 1 Processor -- start 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.007+0000 7fb9c2323700 1 -- start start 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.007+0000 7fb9c2323700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.007+0000 7fb9c2323700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1053e0 con 0x7fb9bc071000 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49660/0 (socket says 172.21.15.134:49660) 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9bbfff700 1 -- 172.21.15.134:0/469495412 learned_addr learned my addr 172.21.15.134:0/469495412 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9bbfff700 1 -- 172.21.15.134:0/469495412 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc105520 con 0x7fb9bc071000 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9bbfff700 1 --2- 172.21.15.134:0/469495412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0098e0 tx=0x7fb9ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160760e724b89005 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9baffd700 1 -- 172.21.15.134:0/469495412 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ac00b860 con 0x7fb9bc071000 2024-01-19T06:19:08.961 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9baffd700 1 -- 172.21.15.134:0/469495412 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9ac00be70 con 0x7fb9bc071000 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.008+0000 7fb9baffd700 1 -- 172.21.15.134:0/469495412 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ac010380 con 0x7fb9bc071000 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 -- 172.21.15.134:0/469495412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 msgr2=0x7fb9bc104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 --2- 172.21.15.134:0/469495412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0098e0 tx=0x7fb9ac009250 comp rx=0 tx=0).stop 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 -- 172.21.15.134:0/469495412 shutdown_connections 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 --2- 172.21.15.134:0/469495412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc104ea0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 -- 172.21.15.134:0/469495412 >> 172.21.15.134:0/469495412 conn(0x7fb9bc0fc970 msgr2=0x7fb9bc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 -- 172.21.15.134:0/469495412 shutdown_connections 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 -- 172.21.15.134:0/469495412 wait complete. 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.009+0000 7fb9c2323700 1 Processor -- start 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9c2323700 1 -- start start 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9c2323700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9c2323700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc111700 con 0x7fb9bc071000 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49664/0 (socket says 172.21.15.134:49664) 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9bbfff700 1 -- 172.21.15.134:0/2786885203 learned_addr learned my addr 172.21.15.134:0/2786885203 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9bbfff700 1 -- 172.21.15.134:0/2786885203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0094f0 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9bbfff700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac003080 tx=0x7fb9ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ac010960 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9ac01a430 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.010+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc111900 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.011+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ac0245b0 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.011+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc111d10 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.011+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fb9ac010ac0 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.012+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc04bb80 con 0x7fb9bc071000 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.012+0000 7fb9b8ff9700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 0x7fb9a4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:08.963 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.012+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb9ac04a200 con 0x7fb9bc071000 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.012+0000 7fb9bb7fe700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 0x7fb9a4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.012+0000 7fb9bb7fe700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 0x7fb9a4038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009990 tx=0x7fb9b0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.016+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb9ac02d3f0 con 0x7fb9bc071000 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.218+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb9bc10ad60 con 0x7fb9a4036640 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.437+0000 7fb9b8ff9700 1 -- 172.21.15.134:0/2786885203 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb9bc10ad60 con 0x7fb9a4036640 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 msgr2=0x7fb9a4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 0x7fb9a4038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009990 tx=0x7fb9b0006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 msgr2=0x7fb9bc1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac003080 tx=0x7fb9ac009250 comp rx=0 tx=0).stop 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 shutdown_connections 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb9a4036640 0x7fb9a4038af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 --2- 172.21.15.134:0/2786885203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc071000 0x7fb9bc1111c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 >> 172.21.15.134:0/2786885203 conn(0x7fb9bc0fc970 msgr2=0x7fb9bc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:08.964 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 shutdown_connections 2024-01-19T06:19:08.965 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:08.440+0000 7fb9c2323700 1 -- 172.21.15.134:0/2786885203 wait complete. 2024-01-19T06:19:08.965 INFO:teuthology.orchestra.run.smithi134.stdout:Fetching dashboard port number... 2024-01-19T06:19:09.596 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:09 smithi134 ceph-mon[28283]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:19:09.596 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:09 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:10.168 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 8443 2024-01-19T06:19:10.168 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.403+0000 7f66d57e7700 1 Processor -- start 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.403+0000 7f66d57e7700 1 -- start start 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66d57e7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66d57e7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d01042e0 con 0x7f66d0103990 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49676/0 (socket says 172.21.15.134:49676) 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66ceffd700 1 -- 172.21.15.134:0/1055321809 learned_addr learned my addr 172.21.15.134:0/1055321809 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66ceffd700 1 -- 172.21.15.134:0/1055321809 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d0104420 con 0x7f66d0103990 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66ceffd700 1 --2- 172.21.15.134:0/1055321809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66b80098e0 tx=0x7f66b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f64a16b51f0022 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66cdffb700 1 -- 172.21.15.134:0/1055321809 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b800b860 con 0x7f66d0103990 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66cdffb700 1 -- 172.21.15.134:0/1055321809 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66b800be70 con 0x7f66d0103990 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.404+0000 7f66cdffb700 1 -- 172.21.15.134:0/1055321809 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b8010380 con 0x7f66d0103990 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 -- 172.21.15.134:0/1055321809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 msgr2=0x7f66d0103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 --2- 172.21.15.134:0/1055321809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66b80098e0 tx=0x7f66b8009250 comp rx=0 tx=0).stop 2024-01-19T06:19:10.169 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 -- 172.21.15.134:0/1055321809 shutdown_connections 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 --2- 172.21.15.134:0/1055321809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0103da0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 -- 172.21.15.134:0/1055321809 >> 172.21.15.134:0/1055321809 conn(0x7f66d00fc0b0 msgr2=0x7f66d00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 -- 172.21.15.134:0/1055321809 shutdown_connections 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.405+0000 7f66d57e7700 1 -- 172.21.15.134:0/1055321809 wait complete. 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66d57e7700 1 Processor -- start 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66d57e7700 1 -- start start 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66d57e7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66d57e7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0111690 con 0x7f66d0103990 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66ceffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49690/0 (socket says 172.21.15.134:49690) 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66ceffd700 1 -- 172.21.15.134:0/932159359 learned_addr learned my addr 172.21.15.134:0/932159359 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66ceffd700 1 -- 172.21.15.134:0/932159359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80094f0 con 0x7f66d0103990 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.406+0000 7f66ceffd700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f66b8011b60 tx=0x7f66b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:10.170 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b8010850 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66b8010e60 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d0111890 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b801ae20 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d0111ca0 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f66b802d3f0 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.407+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d004bb80 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.408+0000 7f66c77fe700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 0x7f66bc038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.408+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f66b804a310 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.408+0000 7f66ce7fc700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 0x7f66bc038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.408+0000 7f66ce7fc700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 0x7f66bc038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009990 tx=0x7f66c0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.412+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f66b802b8e0 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.596+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f66d0100de0 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.597+0000 7f66c77fe700 1 -- 172.21.15.134:0/932159359 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f66b802b2f0 con 0x7f66d0103990 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 msgr2=0x7f66bc038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:10.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 0x7f66bc038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66c0009990 tx=0x7f66c0006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 msgr2=0x7f66d0111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f66b8011b60 tx=0x7f66b8009250 comp rx=0 tx=0).stop 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 shutdown_connections 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f66bc036690 0x7f66bc038b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 --2- 172.21.15.134:0/932159359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66d0103990 0x7f66d0111150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 >> 172.21.15.134:0/932159359 conn(0x7f66d00fc0b0 msgr2=0x7f66d00fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 shutdown_connections 2024-01-19T06:19:10.172 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:09.599+0000 7f66d57e7700 1 -- 172.21.15.134:0/932159359 wait complete. 2024-01-19T06:19:10.175 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T06:19:10.175 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-01-19T06:19:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T06:19:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-01-19T06:19:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-01-19T06:19:10.183 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-19T06:19:10.184 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph Dashboard is now available at: 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: URL: https://smithi134.front.sepia.ceph.com:8443/ 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: User: admin 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: Password: 9y4c476or1 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:Saving cluster configuration to /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config directory 2024-01-19T06:19:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling autotune for osd_memory_target 2024-01-19T06:19:10.559 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:10 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/932159359' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c933e9700 1 Processor -- start 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c933e9700 1 -- start start 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c933e9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c933e9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c105420 con 0x7f2c8c076c20 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c91185700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:11.317 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c91185700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49704/0 (socket says 172.21.15.134:49704) 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.560+0000 7f2c91185700 1 -- 172.21.15.134:0/4164166157 learned_addr learned my addr 172.21.15.134:0/4164166157 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c91185700 1 -- 172.21.15.134:0/4164166157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c105560 con 0x7f2c8c076c20 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c91185700 1 --2- 172.21.15.134:0/4164166157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2c880098e0 tx=0x7f2c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfcbedae0ba92e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c83fff700 1 -- 172.21.15.134:0/4164166157 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8800b860 con 0x7f2c8c076c20 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c83fff700 1 -- 172.21.15.134:0/4164166157 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c8800be70 con 0x7f2c8c076c20 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c83fff700 1 -- 172.21.15.134:0/4164166157 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c88010380 con 0x7f2c8c076c20 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.561+0000 7f2c933e9700 1 -- 172.21.15.134:0/4164166157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 msgr2=0x7f2c8c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 --2- 172.21.15.134:0/4164166157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2c880098e0 tx=0x7f2c88009250 comp rx=0 tx=0).stop 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- 172.21.15.134:0/4164166157 shutdown_connections 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 --2- 172.21.15.134:0/4164166157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c104ee0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- 172.21.15.134:0/4164166157 >> 172.21.15.134:0/4164166157 conn(0x7f2c8c0fc950 msgr2=0x7f2c8c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:11.318 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- 172.21.15.134:0/4164166157 shutdown_connections 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- 172.21.15.134:0/4164166157 wait complete. 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 Processor -- start 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- start start 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.562+0000 7f2c933e9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1116d0 con 0x7f2c8c076c20 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c91185700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c91185700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49720/0 (socket says 172.21.15.134:49720) 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c91185700 1 -- 172.21.15.134:0/3101827666 learned_addr learned my addr 172.21.15.134:0/3101827666 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c91185700 1 -- 172.21.15.134:0/3101827666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c880094f0 con 0x7f2c8c076c20 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c91185700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c880094c0 tx=0x7f2c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c88010850 con 0x7f2c8c076c20 2024-01-19T06:19:11.319 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c88010e60 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c1118d0 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8801a430 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.563+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c111ce0 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.564+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f2c8801a6b0 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.564+0000 7f2c81ffb700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 0x7f2c78038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.564+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8c04bb80 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.564+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2c8804a670 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.564+0000 7f2c90984700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 0x7f2c78038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.565+0000 7f2c90984700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 0x7f2c78038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009990 tx=0x7f2c7c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.569+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2c88018b80 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.752+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2c8c1062e0 con 0x7f2c8c076c20 2024-01-19T06:19:11.320 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.753+0000 7f2c81ffb700 1 -- 172.21.15.134:0/3101827666 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2c88018350 con 0x7f2c8c076c20 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 msgr2=0x7f2c78038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 0x7f2c78038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009990 tx=0x7f2c7c006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 msgr2=0x7f2c8c111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2c880094c0 tx=0x7f2c88009250 comp rx=0 tx=0).stop 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 shutdown_connections 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2c780366e0 0x7f2c78038b90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 --2- 172.21.15.134:0/3101827666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c8c076c20 0x7f2c8c111190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 >> 172.21.15.134:0/3101827666 conn(0x7f2c8c0fc950 msgr2=0x7f2c8c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 shutdown_connections 2024-01-19T06:19:11.321 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:10.755+0000 7f2c933e9700 1 -- 172.21.15.134:0/3101827666 wait complete. 2024-01-19T06:19:11.722 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:11 smithi134 ceph-mon[28283]: mgrmap e13: smithi134.qrxokr(active, since 6s) 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c4253e700 1 Processor -- start 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c4253e700 1 -- start start 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c4253e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c4253e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c104240 con 0x7f7c3c1038f0 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49730/0 (socket says 172.21.15.134:49730) 2024-01-19T06:19:12.557 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.723+0000 7f7c3bfff700 1 -- 172.21.15.134:0/3803162701 learned_addr learned my addr 172.21.15.134:0/3803162701 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.724+0000 7f7c3bfff700 1 -- 172.21.15.134:0/3803162701 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c104380 con 0x7f7c3c1038f0 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.724+0000 7f7c3bfff700 1 --2- 172.21.15.134:0/3803162701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0098e0 tx=0x7f7c2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f907ff1d14896c server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.724+0000 7f7c3affd700 1 -- 172.21.15.134:0/3803162701 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c00b860 con 0x7f7c3c1038f0 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.724+0000 7f7c3affd700 1 -- 172.21.15.134:0/3803162701 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2c00be70 con 0x7f7c3c1038f0 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.724+0000 7f7c3affd700 1 -- 172.21.15.134:0/3803162701 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c010380 con 0x7f7c3c1038f0 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 -- 172.21.15.134:0/3803162701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 msgr2=0x7f7c3c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 --2- 172.21.15.134:0/3803162701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0098e0 tx=0x7f7c2c009250 comp rx=0 tx=0).stop 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 -- 172.21.15.134:0/3803162701 shutdown_connections 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 --2- 172.21.15.134:0/3803162701 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c103d00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 -- 172.21.15.134:0/3803162701 >> 172.21.15.134:0/3803162701 conn(0x7f7c3c0fc0b0 msgr2=0x7f7c3c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 -- 172.21.15.134:0/3803162701 shutdown_connections 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 -- 172.21.15.134:0/3803162701 wait complete. 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.725+0000 7f7c4253e700 1 Processor -- start 2024-01-19T06:19:12.558 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c4253e700 1 -- start start 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c4253e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c4253e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c1116b0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c3bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49740/0 (socket says 172.21.15.134:49740) 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c3bfff700 1 -- 172.21.15.134:0/1680238136 learned_addr learned my addr 172.21.15.134:0/1680238136 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c3bfff700 1 -- 172.21.15.134:0/1680238136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0094f0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c3bfff700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c011b60 tx=0x7f7c2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c010850 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2c010e60 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3c1118b0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c2c01ae20 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.726+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c111cc0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.727+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f7c2c02d3f0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.727+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3c04bb80 con 0x7f7c3c1038f0 2024-01-19T06:19:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.727+0000 7f7c38ff9700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 0x7f7c24038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.727+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7c2c04a3f0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.728+0000 7f7c3b7fe700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 0x7f7c24038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.728+0000 7f7c3b7fe700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 0x7f7c24038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009990 tx=0x7f7c30006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:11.732+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7c2c02b8e0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.018+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f7c3c100da0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.021+0000 7f7c38ff9700 1 -- 172.21.15.134:0/1680238136 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f7c2c02b2f0 con 0x7f7c3c1038f0 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 msgr2=0x7f7c24038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 0x7f7c24038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009990 tx=0x7f7c30006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 msgr2=0x7f7c3c111170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c011b60 tx=0x7f7c2c009250 comp rx=0 tx=0).stop 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 shutdown_connections 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f7c240366e0 0x7f7c24038b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 --2- 172.21.15.134:0/1680238136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c3c1038f0 0x7f7c3c111170 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:12.560 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 >> 172.21.15.134:0/1680238136 conn(0x7f7c3c0fc0b0 msgr2=0x7f7c3c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 shutdown_connections 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr 2024-01-19T06:19:12.023+0000 7f7c4253e700 1 -- 172.21.15.134:0/1680238136 wait complete. 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:Or, if you are only running a single cluster on this host: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout:Please consider enabling telemetry to help improve Ceph: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: ceph telemetry on 2024-01-19T06:19:12.561 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.562 INFO:teuthology.orchestra.run.smithi134.stdout:For more information see: 2024-01-19T06:19:12.562 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.562 INFO:teuthology.orchestra.run.smithi134.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-01-19T06:19:12.562 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:12.562 INFO:teuthology.orchestra.run.smithi134.stdout:Bootstrap complete. 2024-01-19T06:19:12.593 INFO:tasks.cephadm:Fetching config... 2024-01-19T06:19:12.593 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:19:12.593 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-19T06:19:12.608 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-19T06:19:12.609 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:19:12.609 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-19T06:19:12.665 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-19T06:19:12.665 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:19:12.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/keyring of=/dev/stdout 2024-01-19T06:19:12.732 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-19T06:19:12.733 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:19:12.733 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-19T06:19:12.788 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-19T06:19:12.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6qa1rs3zIY2t5mCa09XumiPcjuvDJY/dItmJWn7Ygr0x8xUMly+ZkNj+H2PNHcYIihRlDgnrwfEDLRKOKqLYLCt6OyJ/yfQpkl5riJxx4zJCD2mJMrZsmUpbtxadaTaKQ0epsF6zy8ErxD1QzgDwCnxUdWJ1XKL0becjx65XGC0hyrnEPN6Gwn4dUAekJ+gY55jBjfiWWaYGRmgqb0Jd45OmNFRjCID+M995dKPS18USdRNFpgBcdUEOgr/CJBIdvHMJKf+NkUDPrnhvhB0Ks420z3Y9aS+qCrSaQhKPkN5pkeJd82O6OfSOHJmcHrjca464ek4SMgBJbU3Mn7Eh5b+ZntwHBCBSDX89Ity1tAG9jwu6uDtOpX9qt0veYkzFkyQs6HiayizOBIaxa885oeGoh+xsBbFIqUFf7cDLq33w1cQJSll6zR2098yzLm47pnr4L7agWitz4hm93R/H11+BhFheqrLc22Phk0/iosQa6uoOGPtRo3TKElWnXDWc= ceph-573cd99a-b692-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-19T06:19:12.866 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6qa1rs3zIY2t5mCa09XumiPcjuvDJY/dItmJWn7Ygr0x8xUMly+ZkNj+H2PNHcYIihRlDgnrwfEDLRKOKqLYLCt6OyJ/yfQpkl5riJxx4zJCD2mJMrZsmUpbtxadaTaKQ0epsF6zy8ErxD1QzgDwCnxUdWJ1XKL0becjx65XGC0hyrnEPN6Gwn4dUAekJ+gY55jBjfiWWaYGRmgqb0Jd45OmNFRjCID+M995dKPS18USdRNFpgBcdUEOgr/CJBIdvHMJKf+NkUDPrnhvhB0Ks420z3Y9aS+qCrSaQhKPkN5pkeJd82O6OfSOHJmcHrjca464ek4SMgBJbU3Mn7Eh5b+ZntwHBCBSDX89Ity1tAG9jwu6uDtOpX9qt0veYkzFkyQs6HiayizOBIaxa885oeGoh+xsBbFIqUFf7cDLq33w1cQJSll6zR2098yzLm47pnr4L7agWitz4hm93R/H11+BhFheqrLc22Phk0/iosQa6uoOGPtRo3TKElWnXDWc= ceph-573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:19:12.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6qa1rs3zIY2t5mCa09XumiPcjuvDJY/dItmJWn7Ygr0x8xUMly+ZkNj+H2PNHcYIihRlDgnrwfEDLRKOKqLYLCt6OyJ/yfQpkl5riJxx4zJCD2mJMrZsmUpbtxadaTaKQ0epsF6zy8ErxD1QzgDwCnxUdWJ1XKL0becjx65XGC0hyrnEPN6Gwn4dUAekJ+gY55jBjfiWWaYGRmgqb0Jd45OmNFRjCID+M995dKPS18USdRNFpgBcdUEOgr/CJBIdvHMJKf+NkUDPrnhvhB0Ks420z3Y9aS+qCrSaQhKPkN5pkeJd82O6OfSOHJmcHrjca464ek4SMgBJbU3Mn7Eh5b+ZntwHBCBSDX89Ity1tAG9jwu6uDtOpX9qt0veYkzFkyQs6HiayizOBIaxa885oeGoh+xsBbFIqUFf7cDLq33w1cQJSll6zR2098yzLm47pnr4L7agWitz4hm93R/H11+BhFheqrLc22Phk0/iosQa6uoOGPtRo3TKElWnXDWc= ceph-573cd99a-b692-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-19T06:19:12.925 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6qa1rs3zIY2t5mCa09XumiPcjuvDJY/dItmJWn7Ygr0x8xUMly+ZkNj+H2PNHcYIihRlDgnrwfEDLRKOKqLYLCt6OyJ/yfQpkl5riJxx4zJCD2mJMrZsmUpbtxadaTaKQ0epsF6zy8ErxD1QzgDwCnxUdWJ1XKL0becjx65XGC0hyrnEPN6Gwn4dUAekJ+gY55jBjfiWWaYGRmgqb0Jd45OmNFRjCID+M995dKPS18USdRNFpgBcdUEOgr/CJBIdvHMJKf+NkUDPrnhvhB0Ks420z3Y9aS+qCrSaQhKPkN5pkeJd82O6OfSOHJmcHrjca464ek4SMgBJbU3Mn7Eh5b+ZntwHBCBSDX89Ity1tAG9jwu6uDtOpX9qt0veYkzFkyQs6HiayizOBIaxa885oeGoh+xsBbFIqUFf7cDLq33w1cQJSll6zR2098yzLm47pnr4L7agWitz4hm93R/H11+BhFheqrLc22Phk0/iosQa6uoOGPtRo3TKElWnXDWc= ceph-573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:19:12.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-19T06:19:13.119 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:13 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1680238136' entity='client.admin' 2024-01-19T06:19:13.197 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 -- 172.21.15.134:0/616040150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 msgr2=0x7f34b80ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 --2- 172.21.15.134:0/616040150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80ff420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34a80048c0 tx=0x7f34a8009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 -- 172.21.15.134:0/616040150 shutdown_connections 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 --2- 172.21.15.134:0/616040150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80ff420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 -- 172.21.15.134:0/616040150 >> 172.21.15.134:0/616040150 conn(0x7f34b80f6e50 msgr2=0x7f34b80f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 -- 172.21.15.134:0/616040150 shutdown_connections 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 -- 172.21.15.134:0/616040150 wait complete. 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.665+0000 7f34be129700 1 Processor -- start 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34be129700 1 -- start start 2024-01-19T06:19:15.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34be129700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34be129700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b80fb320 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34b77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34b77fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49760/0 (socket says 172.21.15.134:49760) 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34b77fe700 1 -- 172.21.15.134:0/3156314297 learned_addr learned my addr 172.21.15.134:0/3156314297 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34b77fe700 1 -- 172.21.15.134:0/3156314297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8005040 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f34b77fe700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009040 tx=0x7f34a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.666+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a8009620 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.667+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34a8009780 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.667+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b80fb520 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.667+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b80fba10 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.667+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a801c5b0 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.667+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f34a801c710 con 0x7f34b8066330 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.668+0000 7f349ffff700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 0x7f34a0038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.668+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f34a8049870 con 0x7f34b8066330 2024-01-19T06:19:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.668+0000 7f34b6ffd700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 0x7f34a0038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.668+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a40052e0 con 0x7f34b8066330 2024-01-19T06:19:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.668+0000 7f34b6ffd700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 0x7f34a0038bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009990 tx=0x7f34ac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.673+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f34a8025400 con 0x7f34b8066330 2024-01-19T06:19:15.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.864+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f34a4005b10 con 0x7f34b8066330 2024-01-19T06:19:15.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.867+0000 7f349ffff700 1 -- 172.21.15.134:0/3156314297 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f34a8011b60 con 0x7f34b8066330 2024-01-19T06:19:15.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 msgr2=0x7f34a0038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:15.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 0x7f34a0038bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009990 tx=0x7f34ac006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:15.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 msgr2=0x7f34b80fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:15.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009040 tx=0x7f34a8009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 shutdown_connections 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f34a0036720 0x7f34a0038bd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 --2- 172.21.15.134:0/3156314297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34b8066330 0x7f34b80fcea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 >> 172.21.15.134:0/3156314297 conn(0x7f34b80f6e50 msgr2=0x7f34b80f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 shutdown_connections 2024-01-19T06:19:15.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:15.870+0000 7f34be129700 1 -- 172.21.15.134:0/3156314297 wait complete. 2024-01-19T06:19:17.002 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-19T06:19:17.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-19T06:19:17.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3156314297' entity='client.admin' 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-19T06:19:17.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:16 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:19:17.682 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:19:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:17 smithi134 ceph-mon[28283]: Deploying daemon crash.smithi134 on smithi134 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3286604758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 msgr2=0x7fa3f00fafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3286604758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f00fafe0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00048c0 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3286604758 shutdown_connections 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3286604758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f00fafe0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3286604758 >> 172.21.15.134:0/3286604758 conn(0x7fa3f00f65d0 msgr2=0x7fa3f00f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3286604758 shutdown_connections 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3286604758 wait complete. 2024-01-19T06:19:19.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 Processor -- start 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.485+0000 7fa3f4b59700 1 -- start start 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3f4b59700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3f4b59700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010d2a0 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3ee59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3ee59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49862/0 (socket says 172.21.15.134:49862) 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3ee59c700 1 -- 172.21.15.134:0/3832529607 learned_addr learned my addr 172.21.15.134:0/3832529607 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3ee59c700 1 -- 172.21.15.134:0/3832529607 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0005040 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3ee59c700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000b690 tx=0x7fa3e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e0009650 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3e00097b0 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f010d4a0 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e001c5b0 con 0x7fa3f00fabd0 2024-01-19T06:19:19.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.486+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f010e060 con 0x7fa3f00fabd0 2024-01-19T06:19:19.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.487+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7fa3e001c870 con 0x7fa3f00fabd0 2024-01-19T06:19:19.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.487+0000 7fa3deffd700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 0x7fa3d8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:19.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.487+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa3e0049b00 con 0x7fa3f00fabd0 2024-01-19T06:19:19.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.487+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f005f270 con 0x7fa3f00fabd0 2024-01-19T06:19:19.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.487+0000 7fa3edd9b700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 0x7fa3d8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:19.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.488+0000 7fa3edd9b700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 0x7fa3d8038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009990 tx=0x7fa3e4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:19.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.492+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3e0011db0 con 0x7fa3f00fabd0 2024-01-19T06:19:19.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.682+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3f010eae0 con 0x7fa3d80366d0 2024-01-19T06:19:19.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.686+0000 7fa3deffd700 1 -- 172.21.15.134:0/3832529607 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3f010eae0 con 0x7fa3d80366d0 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 msgr2=0x7fa3d8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 0x7fa3d8038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009990 tx=0x7fa3e4006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 msgr2=0x7fa3f010cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000b690 tx=0x7fa3e0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 shutdown_connections 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fa3d80366d0 0x7fa3d8038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:19.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 --2- 172.21.15.134:0/3832529607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3f00fabd0 0x7fa3f010cd60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:19.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 >> 172.21.15.134:0/3832529607 conn(0x7fa3f00f65d0 msgr2=0x7fa3f00f73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:19.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 shutdown_connections 2024-01-19T06:19:19.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:19.688+0000 7fa3f4b59700 1 -- 172.21.15.134:0/3832529607 wait complete. 2024-01-19T06:19:20.549 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2024-01-19T06:19:20.549 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:19:20.549 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2024-01-19T06:19:20.567 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:19:20.567 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T06:19:20.624 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2024-01-19T06:19:20.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch host add smithi150 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:20.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:20 smithi134 ceph-mon[28283]: Deploying daemon node-exporter.smithi134 on smithi134 2024-01-19T06:19:21.019 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:19:22.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 -- 172.21.15.134:0/496894447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 msgr2=0x7fbba40ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:22.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 --2- 172.21.15.134:0/496894447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba40ff740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb940048c0 tx=0x7fbb94009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 -- 172.21.15.134:0/496894447 shutdown_connections 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 --2- 172.21.15.134:0/496894447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba40ff740 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 -- 172.21.15.134:0/496894447 >> 172.21.15.134:0/496894447 conn(0x7fbba40f6de0 msgr2=0x7fbba40f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 -- 172.21.15.134:0/496894447 shutdown_connections 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.420+0000 7fbba9287700 1 -- 172.21.15.134:0/496894447 wait complete. 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba9287700 1 Processor -- start 2024-01-19T06:19:22.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba9287700 1 -- start start 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba9287700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba9287700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba410cf40 con 0x7fbba40fd360 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49890/0 (socket says 172.21.15.134:49890) 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.421+0000 7fbba2ffd700 1 -- 172.21.15.134:0/2027935921 learned_addr learned my addr 172.21.15.134:0/2027935921 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbba2ffd700 1 -- 172.21.15.134:0/2027935921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb94005040 con 0x7fbba40fd360 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbba2ffd700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009040 tx=0x7fbb94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb94009620 con 0x7fbba40fd360 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba410d140 con 0x7fbba40fd360 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb94009780 con 0x7fbba40fd360 2024-01-19T06:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb9401c5b0 con 0x7fbba40fd360 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.422+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba410dde0 con 0x7fbba40fd360 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.423+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba405f270 con 0x7fbba40fd360 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.423+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7fbb9401c870 con 0x7fbba40fd360 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.423+0000 7fbb8b7fe700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 0x7fbb8c038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.423+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbb94049a20 con 0x7fbba40fd360 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.423+0000 7fbba27fc700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 0x7fbb8c038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:22.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.424+0000 7fbba27fc700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 0x7fbb8c038b90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009990 tx=0x7fbb98006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:22.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.428+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbb94023300 con 0x7fbba40fd360 2024-01-19T06:19:22.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:22.623+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7fbba410e9a0 con 0x7fbb8c0366e0 2024-01-19T06:19:23.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:23 smithi134 ceph-mon[28283]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:19:24.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.853+0000 7fbb8b7fe700 1 -- 172.21.15.134:0/2027935921 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fbba410e9a0 con 0x7fbb8c0366e0 2024-01-19T06:19:24.854 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi150' with addr '172.21.15.150' 2024-01-19T06:19:24.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 msgr2=0x7fbb8c038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 0x7fbb8c038b90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009990 tx=0x7fbb98006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 msgr2=0x7fbba410ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009040 tx=0x7fbb94009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 shutdown_connections 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fbb8c0366e0 0x7fbb8c038b90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 --2- 172.21.15.134:0/2027935921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbba40fd360 0x7fbba410ca00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 >> 172.21.15.134:0/2027935921 conn(0x7fbba40f6de0 msgr2=0x7fbba40f7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 shutdown_connections 2024-01-19T06:19:24.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:24.856+0000 7fbba9287700 1 -- 172.21.15.134:0/2027935921 wait complete. 2024-01-19T06:19:25.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:24 smithi134 ceph-mon[28283]: Deploying cephadm binary to smithi150 2024-01-19T06:19:25.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:24 smithi134 ceph-mon[28283]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:25.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:24 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:25.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch host ls --format=json 2024-01-19T06:19:25.747 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:19:26.124 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: Added host smithi150 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:26.125 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:25 smithi134 ceph-mon[28283]: Deploying daemon alertmanager.smithi134 on smithi134 2024-01-19T06:19:27.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:26 smithi134 ceph-mon[28283]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:27.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- 172.21.15.134:0/1893703902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 msgr2=0x7f2f000fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:27.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 --2- 172.21.15.134:0/1893703902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f000fcc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00048c0 tx=0x7f2ef0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:27.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- 172.21.15.134:0/1893703902 shutdown_connections 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 --2- 172.21.15.134:0/1893703902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f000fcc90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- 172.21.15.134:0/1893703902 >> 172.21.15.134:0/1893703902 conn(0x7f2f000f7f40 msgr2=0x7f2f000fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- 172.21.15.134:0/1893703902 shutdown_connections 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- 172.21.15.134:0/1893703902 wait complete. 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 Processor -- start 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 -- start start 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.714+0000 7f2f04efd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2f04efd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006ff90 con 0x7f2f000fc880 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2efe59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2efe59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53052/0 (socket says 172.21.15.134:53052) 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2efe59c700 1 -- 172.21.15.134:0/319077019 learned_addr learned my addr 172.21.15.134:0/319077019 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2efe59c700 1 -- 172.21.15.134:0/319077019 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0005040 con 0x7f2f000fc880 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2efe59c700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000b690 tx=0x7f2ef0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:27.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef0009650 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ef00097b0 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f00070190 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ef001c5d0 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.715+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f00070680 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.716+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f2ef0013420 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.716+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f00106510 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.716+0000 7f2eeeffd700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 0x7f2ee8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.717+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2ef0049950 con 0x7f2f000fc880 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.717+0000 7f2efdd9b700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 0x7f2ee8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.717+0000 7f2efdd9b700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 0x7f2ee8038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009990 tx=0x7f2ef4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:27.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.721+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2ef0025400 con 0x7f2f000fc880 2024-01-19T06:19:27.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.930+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f0002b250 con 0x7f2ee80366d0 2024-01-19T06:19:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.931+0000 7f2eeeffd700 1 -- 172.21.15.134:0/319077019 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f2f0002b250 con 0x7f2ee80366d0 2024-01-19T06:19:27.930 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:19:27.930 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2024-01-19T06:19:27.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.933+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 msgr2=0x7f2ee8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:27.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.933+0000 7f2f04efd700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 0x7f2ee8038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009990 tx=0x7f2ef4006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.933+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 msgr2=0x7f2f00071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.933+0000 7f2f04efd700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000b690 tx=0x7f2ef0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 shutdown_connections 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2ee80366d0 0x7f2ee8038b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 --2- 172.21.15.134:0/319077019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f000fc880 0x7f2f00071b10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 >> 172.21.15.134:0/319077019 conn(0x7f2f000f7f40 msgr2=0x7f2f000f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 shutdown_connections 2024-01-19T06:19:27.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:27.934+0000 7f2f04efd700 1 -- 172.21.15.134:0/319077019 wait complete. 2024-01-19T06:19:28.949 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-19T06:19:28.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd crush tunables default 2024-01-19T06:19:29.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:28 smithi134 ceph-mon[28283]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:29.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:28 smithi134 ceph-mon[28283]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T06:19:29.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:28 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:29.906 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:19:31.247 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:30 smithi134 ceph-mon[28283]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:33.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:32 smithi134 ceph-mon[28283]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.852+0000 7f2e05b3f700 1 -- 172.21.15.134:0/46987557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 msgr2=0x7f2e000fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.852+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/46987557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e000fcc90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2df00048c0 tx=0x7f2df0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 -- 172.21.15.134:0/46987557 shutdown_connections 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/46987557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e000fcc90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 -- 172.21.15.134:0/46987557 >> 172.21.15.134:0/46987557 conn(0x7f2e000f7f20 msgr2=0x7f2e000fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 -- 172.21.15.134:0/46987557 shutdown_connections 2024-01-19T06:19:33.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 -- 172.21.15.134:0/46987557 wait complete. 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 Processor -- start 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.853+0000 7f2e05b3f700 1 -- start start 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2e05b3f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2e05b3f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0010d010 con 0x7f2e000fc880 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2dff7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2dff7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53072/0 (socket says 172.21.15.134:53072) 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2dff7fe700 1 -- 172.21.15.134:0/1315356415 learned_addr learned my addr 172.21.15.134:0/1315356415 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2dff7fe700 1 -- 172.21.15.134:0/1315356415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0005040 con 0x7f2e000fc880 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2dff7fe700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009330 tx=0x7f2df0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2df00096b0 con 0x7f2e000fc880 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2df0009810 con 0x7f2e000fc880 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0010d210 con 0x7f2e000fc880 2024-01-19T06:19:33.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2df001c5b0 con 0x7f2e000fc880 2024-01-19T06:19:33.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.854+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0010deb0 con 0x7f2e000fc880 2024-01-19T06:19:33.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.855+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f2df001c820 con 0x7f2e000fc880 2024-01-19T06:19:33.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.855+0000 7f2de7fff700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 0x7f2de8038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:33.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.855+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0005f270 con 0x7f2e000fc880 2024-01-19T06:19:33.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.855+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2df0049a80 con 0x7f2e000fc880 2024-01-19T06:19:33.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.856+0000 7f2dfeffd700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 0x7f2de8038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:33.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.856+0000 7f2dfeffd700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 0x7f2de8038bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009990 tx=0x7f2df4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:33.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:33.860+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2df0078a00 con 0x7f2e000fc880 2024-01-19T06:19:34.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.067+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2e0004bb80 con 0x7f2e000fc880 2024-01-19T06:19:34.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.940+0000 7f2de7fff700 1 -- 172.21.15.134:0/1315356415 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2df0023300 con 0x7f2e000fc880 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.943+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 msgr2=0x7f2de8038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.943+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 0x7f2de8038bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2df4009990 tx=0x7f2df4006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 msgr2=0x7f2e0010cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009330 tx=0x7f2df0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 shutdown_connections 2024-01-19T06:19:34.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f2de8036720 0x7f2de8038bd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:34.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 --2- 172.21.15.134:0/1315356415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e000fc880 0x7f2e0010cad0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:34.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 >> 172.21.15.134:0/1315356415 conn(0x7f2e000f7f20 msgr2=0x7f2e000f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:34.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 shutdown_connections 2024-01-19T06:19:34.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:19:34.944+0000 7f2e05b3f700 1 -- 172.21.15.134:0/1315356415 wait complete. 2024-01-19T06:19:34.944 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1315356415' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T06:19:35.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:34 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:35.524 INFO:tasks.cephadm:Adding mon.smithi134 on smithi134 2024-01-19T06:19:35.524 INFO:tasks.cephadm:Adding mon.smithi150 on smithi150 2024-01-19T06:19:35.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch apply mon '2;smithi134:172.21.15.134=smithi134;smithi150:172.21.15.150=smithi150' 2024-01-19T06:19:35.773 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:35.821 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:36.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:35 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T06:19:36.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:35 smithi134 ceph-mon[28283]: Deploying daemon grafana.smithi134 on smithi134 2024-01-19T06:19:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:35 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1315356415' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-19T06:19:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:35 smithi134 ceph-mon[28283]: osdmap e4: 0 total, 0 up, 0 in 2024-01-19T06:19:37.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:36 smithi134 ceph-mon[28283]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 -- 172.21.15.150:0/1029049809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 msgr2=0x7fdf480fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/1029049809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf480fbb80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0048c0 tx=0x7fdf3c009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 -- 172.21.15.150:0/1029049809 shutdown_connections 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/1029049809 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf480fbb80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 -- 172.21.15.150:0/1029049809 >> 172.21.15.150:0/1029049809 conn(0x7fdf480f6e50 msgr2=0x7fdf480f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 -- 172.21.15.150:0/1029049809 shutdown_connections 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 -- 172.21.15.150:0/1029049809 wait complete. 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.523+0000 7fdf4eae4700 1 Processor -- start 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4eae4700 1 -- start start 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4eae4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4eae4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480664a0 con 0x7fdf480fb770 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4c880700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4c880700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:36134/0 (socket says 172.21.15.150:36134) 2024-01-19T06:19:37.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4c880700 1 -- 172.21.15.150:0/2089440952 learned_addr learned my addr 172.21.15.150:0/2089440952 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4c880700 1 -- 172.21.15.150:0/2089440952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c005040 con 0x7fdf480fb770 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.524+0000 7fdf4c880700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009330 tx=0x7fdf3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.525+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3c0096b0 con 0x7fdf480fb770 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.525+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf3c009810 con 0x7fdf480fb770 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.525+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf480666a0 con 0x7fdf480fb770 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.525+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3c01c5b0 con 0x7fdf480fb770 2024-01-19T06:19:37.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.525+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4810ff30 con 0x7fdf480fb770 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7fdf3c013420 con 0x7fdf480fb770 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf48105400 con 0x7fdf480fb770 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf457fa700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 0x7fdf38038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdf3c049a10 con 0x7fdf480fb770 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf47fff700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 0x7fdf38038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:37.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.526+0000 7fdf47fff700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 0x7fdf38038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009990 tx=0x7fdf34006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.531+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdf3c0119c0 con 0x7fdf480fb770 2024-01-19T06:19:37.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.722+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 --> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi134:172.21.15.134=smithi134;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf4802b310 con 0x7fdf380366d0 2024-01-19T06:19:37.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.727+0000 7fdf457fa700 1 -- 172.21.15.150:0/2089440952 <== mgr.14160 v2:172.21.15.134:6800/1934811542 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdf4802b310 con 0x7fdf380366d0 2024-01-19T06:19:37.727 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mon update... 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 msgr2=0x7fdf38038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 0x7fdf38038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009990 tx=0x7fdf34006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 msgr2=0x7fdf4810f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009330 tx=0x7fdf3c009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 shutdown_connections 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fdf380366d0 0x7fdf38038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 --2- 172.21.15.150:0/2089440952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf480fb770 0x7fdf4810f7b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 >> 172.21.15.150:0/2089440952 conn(0x7fdf480f6e50 msgr2=0x7fdf480f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 shutdown_connections 2024-01-19T06:19:37.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:37.729+0000 7fdf4eae4700 1 -- 172.21.15.150:0/2089440952 wait complete. 2024-01-19T06:19:38.313 DEBUG:teuthology.orchestra.run.smithi150:mon.smithi150> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi150.service 2024-01-19T06:19:38.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:38.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:38.590 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:38.637 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:39.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:38 smithi134 ceph-mon[28283]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:39.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:38 smithi134 ceph-mon[28283]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi134:172.21.15.134=smithi134;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:19:39.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:38 smithi134 ceph-mon[28283]: Saving service mon spec with placement smithi134:172.21.15.134=smithi134;smithi150:172.21.15.150=smithi150;count:2 2024-01-19T06:19:39.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:38 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:40.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 -- 172.21.15.150:0/1318868917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 msgr2=0x7f15940faf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:40.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 --2- 172.21.15.150:0/1318868917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f15940faf60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f15840048c0 tx=0x7f1584009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 -- 172.21.15.150:0/1318868917 shutdown_connections 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 --2- 172.21.15.150:0/1318868917 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f15940faf60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 -- 172.21.15.150:0/1318868917 >> 172.21.15.150:0/1318868917 conn(0x7f15940f65d0 msgr2=0x7f15940f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 -- 172.21.15.150:0/1318868917 shutdown_connections 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.085+0000 7f1599f78700 1 -- 172.21.15.150:0/1318868917 wait complete. 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f1599f78700 1 Processor -- start 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f1599f78700 1 -- start start 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f1599f78700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f1599f78700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159410d000 con 0x7f15940fab50 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f15937fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f15937fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:60314/0 (socket says 172.21.15.150:60314) 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.086+0000 7f15937fe700 1 -- 172.21.15.150:0/969779759 learned_addr learned my addr 172.21.15.150:0/969779759 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f15937fe700 1 -- 172.21.15.150:0/969779759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584005040 con 0x7f15940fab50 2024-01-19T06:19:40.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f15937fe700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1584009330 tx=0x7f1584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:40.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1584004e60 con 0x7f15940fab50 2024-01-19T06:19:40.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1584009670 con 0x7f15940fab50 2024-01-19T06:19:40.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159410d200 con 0x7f15940fab50 2024-01-19T06:19:40.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f158401c3f0 con 0x7f15940fab50 2024-01-19T06:19:40.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.087+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159410dea0 con 0x7f15940fab50 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.088+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f158401c6b0 con 0x7f15940fab50 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.088+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159405f270 con 0x7f15940fab50 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.088+0000 7f1590ff9700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 0x7f1580038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.088+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1584049b60 con 0x7f15940fab50 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.088+0000 7f158affd700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 0x7f1580038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:40.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.089+0000 7f158affd700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 0x7f1580038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f157c009990 tx=0x7f157c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:40.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.093+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1584025400 con 0x7f15940fab50 2024-01-19T06:19:40.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:39 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:40.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.353+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f159410e9f0 con 0x7f15940fab50 2024-01-19T06:19:40.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.354+0000 7f1590ff9700 1 -- 172.21.15.150:0/969779759 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1584011b60 con 0x7f15940fab50 2024-01-19T06:19:40.353 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:40.353 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:40.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 msgr2=0x7f1580038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:40.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 0x7f1580038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f157c009990 tx=0x7f157c006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:40.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 msgr2=0x7f159410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:40.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1584009330 tx=0x7f1584009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:40.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 shutdown_connections 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f15800366d0 0x7f1580038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 --2- 172.21.15.150:0/969779759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15940fab50 0x7f159410cac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 >> 172.21.15.150:0/969779759 conn(0x7f15940f65d0 msgr2=0x7f15940f73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 shutdown_connections 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:40.356+0000 7f1599f78700 1 -- 172.21.15.150:0/969779759 wait complete. 2024-01-19T06:19:40.356 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:41.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:40 smithi134 ceph-mon[28283]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:41.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:40 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/969779759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:41.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:41.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:42.102 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:42.159 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:43.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:42 smithi134 ceph-mon[28283]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:43.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- 172.21.15.150:0/2088800544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 msgr2=0x7fb09c0ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:43.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/2088800544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c0ff470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0900048c0 tx=0x7fb090009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:43.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- 172.21.15.150:0/2088800544 shutdown_connections 2024-01-19T06:19:43.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/2088800544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c0ff470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- 172.21.15.150:0/2088800544 >> 172.21.15.150:0/2088800544 conn(0x7fb09c0f6e40 msgr2=0x7fb09c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- 172.21.15.150:0/2088800544 shutdown_connections 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- 172.21.15.150:0/2088800544 wait complete. 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 Processor -- start 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.756+0000 7fb0a2db4700 1 -- start start 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a2db4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a2db4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c10cfc0 con 0x7fb09c065a60 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a0b50700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a0b50700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:60334/0 (socket says 172.21.15.150:60334) 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a0b50700 1 -- 172.21.15.150:0/1013196044 learned_addr learned my addr 172.21.15.150:0/1013196044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a0b50700 1 -- 172.21.15.150:0/1013196044 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090005040 con 0x7fb09c065a60 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.757+0000 7fb0a0b50700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb09000b690 tx=0x7fb090009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb090009650 con 0x7fb09c065a60 2024-01-19T06:19:43.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0900097b0 con 0x7fb09c065a60 2024-01-19T06:19:43.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c10d1c0 con 0x7fb09c065a60 2024-01-19T06:19:43.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb09001c5b0 con 0x7fb09c065a60 2024-01-19T06:19:43.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c10de60 con 0x7fb09c065a60 2024-01-19T06:19:43.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.758+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7fb09001c870 con 0x7fb09c065a60 2024-01-19T06:19:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.759+0000 7fb0997fa700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 0x7fb08c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.759+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c05f270 con 0x7fb09c065a60 2024-01-19T06:19:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.759+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb090049ac0 con 0x7fb09c065a60 2024-01-19T06:19:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.759+0000 7fb09bfff700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 0x7fb08c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.759+0000 7fb09bfff700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 0x7fb08c038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb088009990 tx=0x7fb088006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:43.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:43.764+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb0900119c0 con 0x7fb09c065a60 2024-01-19T06:19:44.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.020+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb09c10e9b0 con 0x7fb09c065a60 2024-01-19T06:19:44.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.021+0000 7fb0997fa700 1 -- 172.21.15.150:0/1013196044 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb090023990 con 0x7fb09c065a60 2024-01-19T06:19:44.021 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:44.021 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 msgr2=0x7fb08c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 0x7fb08c038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb088009990 tx=0x7fb088006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 msgr2=0x7fb09c10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb09000b690 tx=0x7fb090009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 shutdown_connections 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fb08c0366d0 0x7fb08c038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 --2- 172.21.15.150:0/1013196044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb09c065a60 0x7fb09c10ca80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 >> 172.21.15.150:0/1013196044 conn(0x7fb09c0f6e40 msgr2=0x7fb09c0f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 shutdown_connections 2024-01-19T06:19:44.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:44.023+0000 7fb0a2db4700 1 -- 172.21.15.150:0/1013196044 wait complete. 2024-01-19T06:19:44.024 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:45.199 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:44 smithi134 ceph-mon[28283]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:45.199 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:44 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/1013196044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:45.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:45.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:45.876 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:45.922 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:47.120 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:46 smithi134 ceph-mon[28283]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:47.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 -- 172.21.15.150:0/3654477819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 msgr2=0x7facac0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:47.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 --2- 172.21.15.150:0/3654477819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac0fbb80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faca40048c0 tx=0x7faca4009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 -- 172.21.15.150:0/3654477819 shutdown_connections 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 --2- 172.21.15.150:0/3654477819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac0fbb80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 -- 172.21.15.150:0/3654477819 >> 172.21.15.150:0/3654477819 conn(0x7facac0f6e50 msgr2=0x7facac0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 -- 172.21.15.150:0/3654477819 shutdown_connections 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.316+0000 7facb2f4d700 1 -- 172.21.15.150:0/3654477819 wait complete. 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb2f4d700 1 Processor -- start 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb2f4d700 1 -- start start 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb2f4d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb2f4d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac10bec0 con 0x7facac0fb770 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb0ce9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb0ce9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:60352/0 (socket says 172.21.15.150:60352) 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.317+0000 7facb0ce9700 1 -- 172.21.15.150:0/1841845259 learned_addr learned my addr 172.21.15.150:0/1841845259 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7facb0ce9700 1 -- 172.21.15.150:0/1841845259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca4005040 con 0x7facac0fb770 2024-01-19T06:19:47.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7facb0ce9700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faca4009330 tx=0x7faca4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca40096b0 con 0x7facac0fb770 2024-01-19T06:19:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac10c0c0 con 0x7facac0fb770 2024-01-19T06:19:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faca4009810 con 0x7facac0fb770 2024-01-19T06:19:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faca401c5b0 con 0x7facac0fb770 2024-01-19T06:19:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.318+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac10cd60 con 0x7facac0fb770 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7faca401c870 con 0x7facac0fb770 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac05f270 con 0x7facac0fb770 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7faca97fa700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 0x7fac98038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7faca4049ac0 con 0x7facac0fb770 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7facabfff700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 0x7fac98038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.319+0000 7facabfff700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 0x7fac98038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00ad90 tx=0x7fac9c009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.324+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faca404be60 con 0x7facac0fb770 2024-01-19T06:19:47.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.582+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facac10d8b0 con 0x7facac0fb770 2024-01-19T06:19:47.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.583+0000 7faca97fa700 1 -- 172.21.15.150:0/1841845259 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7faca404be60 con 0x7facac0fb770 2024-01-19T06:19:47.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:47.583 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 msgr2=0x7fac98038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 0x7fac98038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00ad90 tx=0x7fac9c009410 comp rx=0 tx=0).stop 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 msgr2=0x7facac10b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faca4009330 tx=0x7faca4009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 shutdown_connections 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fac980366d0 0x7fac98038b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 --2- 172.21.15.150:0/1841845259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facac0fb770 0x7facac10b980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.585+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 >> 172.21.15.150:0/1841845259 conn(0x7facac0f6e50 msgr2=0x7facac0f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.586+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 shutdown_connections 2024-01-19T06:19:47.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:47.586+0000 7facb2f4d700 1 -- 172.21.15.150:0/1841845259 wait complete. 2024-01-19T06:19:47.586 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:48.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/1841845259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:48.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:48.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:48.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:48.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:48.059 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:48.059 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:47 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:49.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:49.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:49.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:48 smithi134 ceph-mon[28283]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:49.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:48 smithi134 ceph-mon[28283]: Deploying daemon prometheus.smithi134 on smithi134 2024-01-19T06:19:49.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:48 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:49.567 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:49.614 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 -- 172.21.15.150:0/3395612959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 msgr2=0x7ffa500faf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3395612959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa500faf40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009c60 tx=0x7ffa48004d10 comp rx=0 tx=0).stop 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 -- 172.21.15.150:0/3395612959 shutdown_connections 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3395612959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa500faf40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 -- 172.21.15.150:0/3395612959 >> 172.21.15.150:0/3395612959 conn(0x7ffa500f65d0 msgr2=0x7ffa500f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 -- 172.21.15.150:0/3395612959 shutdown_connections 2024-01-19T06:19:51.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.112+0000 7ffa5514a700 1 -- 172.21.15.150:0/3395612959 wait complete. 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa5514a700 1 Processor -- start 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa5514a700 1 -- start start 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa5514a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa5514a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5010d0c0 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa4ed9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa4ed9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:52698/0 (socket says 172.21.15.150:52698) 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa4ed9d700 1 -- 172.21.15.150:0/3437574403 learned_addr learned my addr 172.21.15.150:0/3437574403 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa4ed9d700 1 -- 172.21.15.150:0/3437574403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48009940 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.113+0000 7ffa4ed9d700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000f80 tx=0x7ffa48004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa4801d440 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5010d2c0 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa4801da50 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa480129f0 con 0x7ffa500fab30 2024-01-19T06:19:51.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5010df60 con 0x7ffa500fab30 2024-01-19T06:19:51.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7ffa48019040 con 0x7ffa500fab30 2024-01-19T06:19:51.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.114+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5005f270 con 0x7ffa500fab30 2024-01-19T06:19:51.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.115+0000 7ffa477fe700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 0x7ffa3c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:51.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.115+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ffa4804a0f0 con 0x7ffa500fab30 2024-01-19T06:19:51.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.115+0000 7ffa4e59c700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 0x7ffa3c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:51.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.115+0000 7ffa4e59c700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 0x7ffa3c038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009990 tx=0x7ffa38006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:51.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.119+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffa4802eb00 con 0x7ffa500fab30 2024-01-19T06:19:51.220 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:50 smithi134 ceph-mon[28283]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:51.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.374+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa5010eab0 con 0x7ffa500fab30 2024-01-19T06:19:51.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.374+0000 7ffa477fe700 1 -- 172.21.15.150:0/3437574403 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7ffa48021090 con 0x7ffa500fab30 2024-01-19T06:19:51.374 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:51.374 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 msgr2=0x7ffa3c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 0x7ffa3c038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009990 tx=0x7ffa38006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 msgr2=0x7ffa5010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000f80 tx=0x7ffa48004d10 comp rx=0 tx=0).stop 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 shutdown_connections 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7ffa3c0366d0 0x7ffa3c038b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 --2- 172.21.15.150:0/3437574403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa500fab30 0x7ffa5010cb80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 >> 172.21.15.150:0/3437574403 conn(0x7ffa500f65d0 msgr2=0x7ffa500f74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 shutdown_connections 2024-01-19T06:19:51.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:51.376+0000 7ffa5514a700 1 -- 172.21.15.150:0/3437574403 wait complete. 2024-01-19T06:19:51.377 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:52.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:51 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/3437574403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:52.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:52.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:53.185 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:53.324 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:52 smithi134 ceph-mon[28283]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 -- 172.21.15.150:0/714699501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 msgr2=0x7f53000fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 --2- 172.21.15.150:0/714699501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000fbb80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f52f00048c0 tx=0x7f52f0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 -- 172.21.15.150:0/714699501 shutdown_connections 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 --2- 172.21.15.150:0/714699501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000fbb80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 -- 172.21.15.150:0/714699501 >> 172.21.15.150:0/714699501 conn(0x7f53000f6e50 msgr2=0x7f53000f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 -- 172.21.15.150:0/714699501 shutdown_connections 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.575+0000 7f5307f72700 1 -- 172.21.15.150:0/714699501 wait complete. 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5307f72700 1 Processor -- start 2024-01-19T06:19:54.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5307f72700 1 -- start start 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5307f72700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5307f72700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53000f73c0 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5305d0e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5305d0e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:52726/0 (socket says 172.21.15.150:52726) 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5305d0e700 1 -- 172.21.15.150:0/858893602 learned_addr learned my addr 172.21.15.150:0/858893602 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5305d0e700 1 -- 172.21.15.150:0/858893602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f0005040 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f5305d0e700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009330 tx=0x7f52f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.576+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f00096b0 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52f0009810 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53000f75c0 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52f001c5b0 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53000f7ab0 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7f52f0013420 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.577+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5300106410 con 0x7f53000fb770 2024-01-19T06:19:54.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.578+0000 7f52f67fc700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 0x7f52ec038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:54.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.578+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f52f0049a70 con 0x7f53000fb770 2024-01-19T06:19:54.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.578+0000 7f530550d700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 0x7f52ec038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:54.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.578+0000 7f530550d700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 0x7f52ec038bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009990 tx=0x7f52fc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:54.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.582+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f52f00119c0 con 0x7f53000fb770 2024-01-19T06:19:54.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.841+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f530004bb80 con 0x7f53000fb770 2024-01-19T06:19:54.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.842+0000 7f52f67fc700 1 -- 172.21.15.150:0/858893602 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f52f0011370 con 0x7f53000fb770 2024-01-19T06:19:54.841 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:54.842 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 msgr2=0x7f52ec038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 0x7f52ec038bd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009990 tx=0x7f52fc006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 msgr2=0x7f53000f8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009330 tx=0x7f52f0009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 shutdown_connections 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f52ec036720 0x7f52ec038bd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 --2- 172.21.15.150:0/858893602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53000fb770 0x7f53000f8f40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 >> 172.21.15.150:0/858893602 conn(0x7f53000f6e50 msgr2=0x7f53000660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 shutdown_connections 2024-01-19T06:19:54.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:54.844+0000 7f5307f72700 1 -- 172.21.15.150:0/858893602 wait complete. 2024-01-19T06:19:54.845 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:55.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:54 smithi134 ceph-mon[28283]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:55.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:54 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/858893602' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:56.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:56.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:19:56.667 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:56.714 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:19:57.026 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:56 smithi134 ceph-mon[28283]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:58.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 -- 172.21.15.150:0/1535664292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 msgr2=0x7fddd80fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:58.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 --2- 172.21.15.150:0/1535664292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80fbb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fddc80048c0 tx=0x7fddc8009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:58.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 -- 172.21.15.150:0/1535664292 shutdown_connections 2024-01-19T06:19:58.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 --2- 172.21.15.150:0/1535664292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80fbb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:58.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 -- 172.21.15.150:0/1535664292 >> 172.21.15.150:0/1535664292 conn(0x7fddd80f6e50 msgr2=0x7fddd80f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:58.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 -- 172.21.15.150:0/1535664292 shutdown_connections 2024-01-19T06:19:58.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 -- 172.21.15.150:0/1535664292 wait complete. 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.128+0000 7fddde5bb700 1 Processor -- start 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddde5bb700 1 -- start start 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddde5bb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddde5bb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80f73a0 con 0x7fddd80fb770 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddd7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddd7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:52746/0 (socket says 172.21.15.150:52746) 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddd7fff700 1 -- 172.21.15.150:0/2110146288 learned_addr learned my addr 172.21.15.150:0/2110146288 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:19:58.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddd7fff700 1 -- 172.21.15.150:0/2110146288 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8005040 con 0x7fddd80fb770 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.129+0000 7fddd7fff700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009330 tx=0x7fddc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.130+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc80096b0 con 0x7fddd80fb770 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.130+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd80f75a0 con 0x7fddd80fb770 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.130+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddc8009810 con 0x7fddd80fb770 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.130+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc801c5b0 con 0x7fddd80fb770 2024-01-19T06:19:58.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.130+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd80f7a90 con 0x7fddd80fb770 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 13) v1 ==== 43338+0+0 (secure 0 0 0) 0x7fddc801c870 con 0x7fddd80fb770 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddd4ff9700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 0x7fddc0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fddc8049a00 con 0x7fddd80fb770 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd8106410 con 0x7fddd80fb770 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddd77fe700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 0x7fddc0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:19:58.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.131+0000 7fddd77fe700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 0x7fddc0038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009990 tx=0x7fddcc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:19:58.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.136+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fddc8023300 con 0x7fddd80fb770 2024-01-19T06:19:58.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.392+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddd802b110 con 0x7fddd80fb770 2024-01-19T06:19:58.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.392+0000 7fddd4ff9700 1 -- 172.21.15.150:0/2110146288 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fddc8011b60 con 0x7fddd80fb770 2024-01-19T06:19:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:19:58.392 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 msgr2=0x7fddc0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 0x7fddc0038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009990 tx=0x7fddcc006e40 comp rx=0 tx=0).stop 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 msgr2=0x7fddd80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009330 tx=0x7fddc8009e30 comp rx=0 tx=0).stop 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 shutdown_connections 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7fddc00366d0 0x7fddc0038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 --2- 172.21.15.150:0/2110146288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fddd80fb770 0x7fddd80f8f20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 >> 172.21.15.150:0/2110146288 conn(0x7fddd80f6e50 msgr2=0x7fddd80660c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:19:58.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 shutdown_connections 2024-01-19T06:19:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:19:58.395+0000 7fddde5bb700 1 -- 172.21.15.150:0/2110146288 wait complete. 2024-01-19T06:19:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:19:59.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:58 smithi134 ceph-mon[28283]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:19:59.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:58 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2110146288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:19:59.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:58 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:59.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:58 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:19:59.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:19:59.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:00.242 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:00.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:59 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:19:59 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-19T06:20:01.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:00 smithi134 ceph-mon[28283]: from='mgr.14160 172.21.15.134:0/1682829847' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-19T06:20:01.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:00 smithi134 ceph-mon[28283]: mgrmap e14: smithi134.qrxokr(active, since 56s) 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- 172.21.15.150:0/501791592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 msgr2=0x7f62a00fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 --2- 172.21.15.150:0/501791592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a00fe300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f62900048c0 tx=0x7f6290009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- 172.21.15.150:0/501791592 shutdown_connections 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 --2- 172.21.15.150:0/501791592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a00fe300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- 172.21.15.150:0/501791592 >> 172.21.15.150:0/501791592 conn(0x7f62a00f65d0 msgr2=0x7f62a00f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- 172.21.15.150:0/501791592 shutdown_connections 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- 172.21.15.150:0/501791592 wait complete. 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 Processor -- start 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.662+0000 7f62a48f8700 1 -- start start 2024-01-19T06:20:01.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f62a48f8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f62a48f8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a010d070 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f629e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f629e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:36914/0 (socket says 172.21.15.150:36914) 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f629e59c700 1 -- 172.21.15.150:0/3800814238 learned_addr learned my addr 172.21.15.150:0/3800814238 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f629e59c700 1 -- 172.21.15.150:0/3800814238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290005040 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f629e59c700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6290009f80 tx=0x7f6290009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629000b5c0 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a010d270 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6290004e60 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a010df10 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.663+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f629001c550 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 14) v1 ==== 43352+0+0 (secure 0 0 0) 0x7f62900123c0 con 0x7f62a00fdef0 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f628effd700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 0x7f6288038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:01.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6290049220 con 0x7f62a00fdef0 2024-01-19T06:20:01.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f629dd9b700 1 -- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 msgr2=0x7f6288038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/1934811542 2024-01-19T06:20:01.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f629dd9b700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 0x7f6288038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:20:01.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.664+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a005f270 con 0x7f62a00fdef0 2024-01-19T06:20:01.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.669+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6290021020 con 0x7f62a00fdef0 2024-01-19T06:20:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.865+0000 7f629dd9b700 1 -- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 msgr2=0x7f6288038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/1934811542 2024-01-19T06:20:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.865+0000 7f629dd9b700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 0x7f6288038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:20:01.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.924+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62a004bb80 con 0x7f62a00fdef0 2024-01-19T06:20:01.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.925+0000 7f628effd700 1 -- 172.21.15.150:0/3800814238 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f62900127b0 con 0x7f62a00fdef0 2024-01-19T06:20:01.924 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:01.924 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 msgr2=0x7f6288038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 0x7f6288038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 msgr2=0x7f62a010cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6290009f80 tx=0x7f6290009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 shutdown_connections 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f6288036720 0x7f6288038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 --2- 172.21.15.150:0/3800814238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62a00fdef0 0x7f62a010cb30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 >> 172.21.15.150:0/3800814238 conn(0x7f62a00f65d0 msgr2=0x7f62a00f73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 shutdown_connections 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:01.927+0000 7f62a48f8700 1 -- 172.21.15.150:0/3800814238 wait complete. 2024-01-19T06:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:02.217 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:01 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/3800814238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:03.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:03.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:03.779 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:03.826 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:05.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- 172.21.15.150:0/3785565513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 msgr2=0x7f0a240ff4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:05.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/3785565513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a240ff4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a180048c0 tx=0x7f0a18009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:05.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- 172.21.15.150:0/3785565513 shutdown_connections 2024-01-19T06:20:05.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/3785565513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a240ff4b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:05.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- 172.21.15.150:0/3785565513 >> 172.21.15.150:0/3785565513 conn(0x7f0a240f6ec0 msgr2=0x7f0a240f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- 172.21.15.150:0/3785565513 shutdown_connections 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- 172.21.15.150:0/3785565513 wait complete. 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 Processor -- start 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.306+0000 7f0a2b10f700 1 -- start start 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a2b10f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a2b10f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2410cff0 con 0x7f0a240664d0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a28eab700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a28eab700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:36926/0 (socket says 172.21.15.150:36926) 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a28eab700 1 -- 172.21.15.150:0/885370157 learned_addr learned my addr 172.21.15.150:0/885370157 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a28eab700 1 -- 172.21.15.150:0/885370157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18005040 con 0x7f0a240664d0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a28eab700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009330 tx=0x7f0a18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.307+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a180096b0 con 0x7f0a240664d0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2410d1f0 con 0x7f0a240664d0 2024-01-19T06:20:05.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a18009810 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a1801c5b0 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2410de90 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 14) v1 ==== 43352+0+0 (secure 0 0 0) 0x7f0a1801c820 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.308+0000 7f0a217fa700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 0x7f0a14038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.309+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0a18049ab0 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.309+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2405f270 con 0x7f0a240664d0 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.309+0000 7f0a23fff700 1 -- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 msgr2=0x7f0a14038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/1934811542 2024-01-19T06:20:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.309+0000 7f0a23fff700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 0x7f0a14038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:20:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.314+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0a18023300 con 0x7f0a240664d0 2024-01-19T06:20:05.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.510+0000 7f0a23fff700 1 -- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 msgr2=0x7f0a14038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.134:6800/1934811542 2024-01-19T06:20:05.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.510+0000 7f0a23fff700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 0x7f0a14038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T06:20:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.573+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a2410e940 con 0x7f0a240664d0 2024-01-19T06:20:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.574+0000 7f0a217fa700 1 -- 172.21.15.150:0/885370157 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0a18011b60 con 0x7f0a240664d0 2024-01-19T06:20:05.573 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:05.573 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.575+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 msgr2=0x7f0a14038c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.575+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 0x7f0a14038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.575+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 msgr2=0x7f0a2410cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.575+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009330 tx=0x7f0a18009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 shutdown_connections 2024-01-19T06:20:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:6800/1934811542,v1:172.21.15.134:6801/1934811542] conn(0x7f0a14036770 0x7f0a14038c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:05.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 --2- 172.21.15.150:0/885370157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a240664d0 0x7f0a2410cab0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:05.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 >> 172.21.15.150:0/885370157 conn(0x7f0a240f6ec0 msgr2=0x7f0a240f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:05.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 shutdown_connections 2024-01-19T06:20:05.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:05.576+0000 7f0a2b10f700 1 -- 172.21.15.150:0/885370157 wait complete. 2024-01-19T06:20:05.576 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:05.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:05 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/885370157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: Active manager daemon smithi134.qrxokr restarted 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: Activating manager daemon smithi134.qrxokr 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: mgrmap e15: smithi134.qrxokr(active, starting, since 0.00304739s) 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.qrxokr", "id": "smithi134.qrxokr"}]: dispatch 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T06:20:06.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: Manager daemon smithi134.qrxokr is now available 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/mirror_snapshot_schedule"}]: dispatch 2024-01-19T06:20:06.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.qrxokr/trash_purge_schedule"}]: dispatch 2024-01-19T06:20:07.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:07.165 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:07.413 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:07.460 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:08.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: [19/Jan/2024:06:20:06] ENGINE Bus STARTING 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: [19/Jan/2024:06:20:06] ENGINE Serving on https://172.21.15.134:7150 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: [19/Jan/2024:06:20:06] ENGINE Bus STARTED 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:08.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:07 smithi134 ceph-mon[28283]: mgrmap e16: smithi134.qrxokr(active, since 1.00534s) 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 -- 172.21.15.150:0/2546283179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 msgr2=0x7fdd3c0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2546283179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c0fbb80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0048c0 tx=0x7fdd2c009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 -- 172.21.15.150:0/2546283179 shutdown_connections 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2546283179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c0fbb80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 -- 172.21.15.150:0/2546283179 >> 172.21.15.150:0/2546283179 conn(0x7fdd3c0f6e50 msgr2=0x7fdd3c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 -- 172.21.15.150:0/2546283179 shutdown_connections 2024-01-19T06:20:08.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.838+0000 7fdd4163b700 1 -- 172.21.15.150:0/2546283179 wait complete. 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.839+0000 7fdd4163b700 1 Processor -- start 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.839+0000 7fdd4163b700 1 -- start start 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.839+0000 7fdd4163b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.839+0000 7fdd4163b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c10f220 con 0x7fdd3c0fb770 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd3affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd3affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:45348/0 (socket says 172.21.15.150:45348) 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd3affd700 1 -- 172.21.15.150:0/2634475650 learned_addr learned my addr 172.21.15.150:0/2634475650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd3affd700 1 -- 172.21.15.150:0/2634475650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c005040 con 0x7fdd3c0fb770 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd3affd700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009330 tx=0x7fdd2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd2c0096b0 con 0x7fdd3c0fb770 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd2c009810 con 0x7fdd3c0fb770 2024-01-19T06:20:08.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c10f420 con 0x7fdd3c0fb770 2024-01-19T06:20:08.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd2c01c5b0 con 0x7fdd3c0fb770 2024-01-19T06:20:08.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.840+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c10ff80 con 0x7fdd3c0fb770 2024-01-19T06:20:08.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.841+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 16) v1 ==== 43202+0+0 (secure 0 0 0) 0x7fdd2c01c820 con 0x7fdd3c0fb770 2024-01-19T06:20:08.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.841+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c04bb80 con 0x7fdd3c0fb770 2024-01-19T06:20:08.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.841+0000 7fdd237fe700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 0x7fdd24038a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:08.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.841+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fdd2c049900 con 0x7fdd3c0fb770 2024-01-19T06:20:08.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.842+0000 7fdd3a7fc700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 0x7fdd24038a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:08.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.842+0000 7fdd3a7fc700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 0x7fdd24038a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009990 tx=0x7fdd30006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:08.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:08.846+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdd2c023300 con 0x7fdd3c0fb770 2024-01-19T06:20:09.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.125+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd3c110e00 con 0x7fdd3c0fb770 2024-01-19T06:20:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.125+0000 7fdd237fe700 1 -- 172.21.15.150:0/2634475650 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fdd2c011b60 con 0x7fdd3c0fb770 2024-01-19T06:20:09.125 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:09.125 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:09.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.127+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 msgr2=0x7fdd24038a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:09.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.127+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 0x7fdd24038a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009990 tx=0x7fdd30006e40 comp rx=0 tx=0).stop 2024-01-19T06:20:09.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.127+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 msgr2=0x7fdd3c10ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:09.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.127+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009330 tx=0x7fdd2c009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:09.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 shutdown_connections 2024-01-19T06:20:09.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdd240365b0 0x7fdd24038a60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:09.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 --2- 172.21.15.150:0/2634475650 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd3c0fb770 0x7fdd3c10ece0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:09.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 >> 172.21.15.150:0/2634475650 conn(0x7fdd3c0f6e50 msgr2=0x7fdd3c0660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:09.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 shutdown_connections 2024-01-19T06:20:09.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:09.128+0000 7fdd4163b700 1 -- 172.21.15.150:0/2634475650 wait complete. 2024-01-19T06:20:09.129 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:09.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:09.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:09 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2634475650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:10.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:10 smithi134 ceph-mon[28283]: mgrmap e17: smithi134.qrxokr(active, since 2s) 2024-01-19T06:20:10.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:10.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:11.231 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:11.291 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T06:20:11.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:11.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:11.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:11.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:11.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:20:13.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1523799669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 msgr2=0x7f2d34102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1523799669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34102ca0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2d300048c0 tx=0x7f2d30009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1523799669 shutdown_connections 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1523799669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34102ca0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1523799669 >> 172.21.15.150:0/1523799669 conn(0x7f2d340f8410 msgr2=0x7f2d340fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1523799669 shutdown_connections 2024-01-19T06:20:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.412+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1523799669 wait complete. 2024-01-19T06:20:13.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3c6fd700 1 Processor -- start 2024-01-19T06:20:13.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3c6fd700 1 -- start start 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3c6fd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3c6fd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341130a0 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3a499700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3a499700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:45366/0 (socket says 172.21.15.150:45366) 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3a499700 1 -- 172.21.15.150:0/1215889635 learned_addr learned my addr 172.21.15.150:0/1215889635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.413+0000 7f2d3a499700 1 -- 172.21.15.150:0/1215889635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30005040 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d3a499700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009300 tx=0x7f2d30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d30004e60 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d341132a0 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d30013420 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3410ca20 con 0x7f2d341028d0 2024-01-19T06:20:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.414+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3001c5b0 con 0x7f2d341028d0 2024-01-19T06:20:13.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.415+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) v1 ==== 43396+0+0 (secure 0 0 0) 0x7f2d3001c7f0 con 0x7f2d341028d0 2024-01-19T06:20:13.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.415+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3405f270 con 0x7f2d341028d0 2024-01-19T06:20:13.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.415+0000 7f2d2affd700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 0x7f2d20038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:13.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.415+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2d30049a30 con 0x7f2d341028d0 2024-01-19T06:20:13.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.415+0000 7f2d39c98700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 0x7f2d20038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:13.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.416+0000 7f2d39c98700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 0x7f2d20038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009990 tx=0x7f2d24006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.420+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2d30011550 con 0x7f2d341028d0 2024-01-19T06:20:13.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: Updating smithi134:/etc/ceph/ceph.conf 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: Updating smithi150:/etc/ceph/ceph.conf 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: Updating smithi150:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.689+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d3410e140 con 0x7f2d341028d0 2024-01-19T06:20:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.690+0000 7f2d2affd700 1 -- 172.21.15.150:0/1215889635 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2d30011370 con 0x7f2d341028d0 2024-01-19T06:20:13.690 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:13.690 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 msgr2=0x7f2d20038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 0x7f2d20038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009990 tx=0x7f2d24006e40 comp rx=0 tx=0).stop 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 msgr2=0x7f2d34112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009300 tx=0x7f2d30009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 shutdown_connections 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2d200367c0 0x7f2d20038c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 --2- 172.21.15.150:0/1215889635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d341028d0 0x7f2d34112b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 >> 172.21.15.150:0/1215889635 conn(0x7f2d340f8410 msgr2=0x7f2d340fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 shutdown_connections 2024-01-19T06:20:13.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:13.692+0000 7f2d3c6fd700 1 -- 172.21.15.150:0/1215889635 wait complete. 2024-01-19T06:20:13.693 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:14.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: Updating smithi150:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: Updating smithi134:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/1215889635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: Updating smithi134:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:14.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:14 smithi134 ceph-mon[28283]: Deploying daemon crash.smithi150 on smithi150 2024-01-19T06:20:15.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:15.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 -- 172.21.15.150:0/3762311642 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 msgr2=0x7f55e40fdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/3762311642 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e40fdc60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009c60 tx=0x7f55d4004d10 comp rx=0 tx=0).stop 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 -- 172.21.15.150:0/3762311642 shutdown_connections 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/3762311642 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e40fdc60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 -- 172.21.15.150:0/3762311642 >> 172.21.15.150:0/3762311642 conn(0x7f55e40f72f0 msgr2=0x7f55e40f9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 -- 172.21.15.150:0/3762311642 shutdown_connections 2024-01-19T06:20:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.296+0000 7f55e9ce9700 1 -- 172.21.15.150:0/3762311642 wait complete. 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e9ce9700 1 Processor -- start 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e9ce9700 1 -- start start 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e9ce9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e9ce9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e4113220 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e37fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e37fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:45374/0 (socket says 172.21.15.150:45374) 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.297+0000 7f55e37fe700 1 -- 172.21.15.150:0/2281940931 learned_addr learned my addr 172.21.15.150:0/2281940931 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55e37fe700 1 -- 172.21.15.150:0/2281940931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d4009940 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55e37fe700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009fd0 tx=0x7f55d4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55d400be40 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e4113420 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55d401d8f0 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55d4012800 con 0x7f55e40fd890 2024-01-19T06:20:17.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.298+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e4113890 con 0x7f55e40fd890 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.299+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) v1 ==== 43396+0+0 (secure 0 0 0) 0x7f55d4019040 con 0x7f55e40fd890 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.299+0000 7f55cbfff700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 0x7f55cc038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.299+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f55d404b150 con 0x7f55e40fd890 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.299+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e405f270 con 0x7f55e40fd890 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.299+0000 7f55e2ffd700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 0x7f55cc038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:17.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.300+0000 7f55e2ffd700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 0x7f55cc038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009990 tx=0x7f55d8006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:17.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.304+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f55d4021080 con 0x7f55e40fd890 2024-01-19T06:20:17.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.573+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55e410e2c0 con 0x7f55e40fd890 2024-01-19T06:20:17.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.573+0000 7f55cbfff700 1 -- 172.21.15.150:0/2281940931 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f55d4017020 con 0x7f55e40fd890 2024-01-19T06:20:17.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:17.574 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 msgr2=0x7f55cc038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 0x7f55cc038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009990 tx=0x7f55d8006e40 comp rx=0 tx=0).stop 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 msgr2=0x7f55e4112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009fd0 tx=0x7f55d4004d10 comp rx=0 tx=0).stop 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 shutdown_connections 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f55cc0367c0 0x7f55cc038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:17.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 --2- 172.21.15.150:0/2281940931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55e40fd890 0x7f55e4112ce0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:17.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 >> 172.21.15.150:0/2281940931 conn(0x7f55e40f72f0 msgr2=0x7f55e4101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:17.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 shutdown_connections 2024-01-19T06:20:17.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:17.576+0000 7f55e9ce9700 1 -- 172.21.15.150:0/2281940931 wait complete. 2024-01-19T06:20:17.577 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:17 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2281940931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:19.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:19.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:19.404 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:19.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:18 smithi134 ceph-mon[28283]: Deploying daemon node-exporter.smithi150 on smithi150 2024-01-19T06:20:22.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.037+0000 7fefc7a6b700 1 -- 172.21.15.150:0/2949458538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 msgr2=0x7fefc0102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:22.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.037+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/2949458538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0102ca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009c60 tx=0x7fefb8004d10 comp rx=0 tx=0).stop 2024-01-19T06:20:22.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.037+0000 7fefc7a6b700 1 -- 172.21.15.150:0/2949458538 shutdown_connections 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.037+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/2949458538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0102ca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.037+0000 7fefc7a6b700 1 -- 172.21.15.150:0/2949458538 >> 172.21.15.150:0/2949458538 conn(0x7fefc00f8410 msgr2=0x7fefc00fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 -- 172.21.15.150:0/2949458538 shutdown_connections 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 -- 172.21.15.150:0/2949458538 wait complete. 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 Processor -- start 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 -- start start 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc7a6b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc0113160 con 0x7fefc01028d0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc5807700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc5807700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:54002/0 (socket says 172.21.15.150:54002) 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.038+0000 7fefc5807700 1 -- 172.21.15.150:0/3901326032 learned_addr learned my addr 172.21.15.150:0/3901326032 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefc5807700 1 -- 172.21.15.150:0/3901326032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8009940 con 0x7fefc01028d0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefc5807700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009fd0 tx=0x7fefb8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb800be40 con 0x7fefc01028d0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc0113360 con 0x7fefc01028d0 2024-01-19T06:20:22.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefb801d8f0 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb8012800 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.039+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc0113850 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.040+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) v1 ==== 43396+0+0 (secure 0 0 0) 0x7fefb8019040 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.040+0000 7fefbe7fc700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 0x7fefa8038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.040+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fefb804a270 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.040+0000 7fefc5006700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 0x7fefa8038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.041+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc005f270 con 0x7fefc01028d0 2024-01-19T06:20:22.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.041+0000 7fefc5006700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 0x7fefa8038cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009990 tx=0x7fefb4006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:22.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.046+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fefb8021080 con 0x7fefc01028d0 2024-01-19T06:20:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.331+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefc010e2a0 con 0x7fefc01028d0 2024-01-19T06:20:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.331+0000 7fefbe7fc700 1 -- 172.21.15.150:0/3901326032 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fefb8017020 con 0x7fefc01028d0 2024-01-19T06:20:22.331 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:22.331 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.333+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 msgr2=0x7fefa8038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.333+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 0x7fefa8038cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009990 tx=0x7fefb4006e40 comp rx=0 tx=0).stop 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.333+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 msgr2=0x7fefc0112c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.333+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009fd0 tx=0x7fefb8004d10 comp rx=0 tx=0).stop 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.333+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 shutdown_connections 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.334+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fefa8036810 0x7fefa8038cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.334+0000 7fefc7a6b700 1 --2- 172.21.15.150:0/3901326032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefc01028d0 0x7fefc0112c20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.334+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 >> 172.21.15.150:0/3901326032 conn(0x7fefc00f8410 msgr2=0x7fefc00f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.334+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 shutdown_connections 2024-01-19T06:20:22.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:22.334+0000 7fefc7a6b700 1 -- 172.21.15.150:0/3901326032 wait complete. 2024-01-19T06:20:22.334 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.xkxrco", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.xkxrco", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:22.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: Deploying daemon mgr.smithi150.xkxrco on smithi150 2024-01-19T06:20:22.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:22 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/3901326032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:24.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:24.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:24.747 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:25 smithi134 ceph-mon[28283]: Deploying daemon mon.smithi150 on smithi150 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 -- 172.21.15.150:0/3678822401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 msgr2=0x7f1aa8101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 --2- 172.21.15.150:0/3678822401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8101ba0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1a980048c0 tx=0x7f1a98009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 -- 172.21.15.150:0/3678822401 shutdown_connections 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 --2- 172.21.15.150:0/3678822401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8101ba0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 -- 172.21.15.150:0/3678822401 >> 172.21.15.150:0/3678822401 conn(0x7f1aa80f7310 msgr2=0x7f1aa80f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 -- 172.21.15.150:0/3678822401 shutdown_connections 2024-01-19T06:20:26.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.188+0000 7f1aad154700 1 -- 172.21.15.150:0/3678822401 wait complete. 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.189+0000 7f1aad154700 1 Processor -- start 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.189+0000 7f1aad154700 1 -- start start 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.189+0000 7f1aad154700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.189+0000 7f1aad154700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa8111fb0 con 0x7f1aa81017d0 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aa6d9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aa6d9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.150:54038/0 (socket says 172.21.15.150:54038) 2024-01-19T06:20:26.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aa6d9d700 1 -- 172.21.15.150:0/2875684110 learned_addr learned my addr 172.21.15.150:0/2875684110 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aa6d9d700 1 -- 172.21.15.150:0/2875684110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98005040 con 0x7f1aa81017d0 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aa6d9d700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009330 tx=0x7f1a98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a98004e60 con 0x7f1aa81017d0 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa81121b0 con 0x7f1aa81017d0 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a980098c0 con 0x7f1aa81017d0 2024-01-19T06:20:26.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.190+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a98013740 con 0x7f1aa81017d0 2024-01-19T06:20:26.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.191+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa8112640 con 0x7f1aa81017d0 2024-01-19T06:20:26.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.191+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) v1 ==== 43396+0+0 (secure 0 0 0) 0x7f1a98013a00 con 0x7f1aa81017d0 2024-01-19T06:20:26.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.191+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa805f270 con 0x7f1aa81017d0 2024-01-19T06:20:26.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.192+0000 7f1a8f7fe700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 0x7f1a90038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:26.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.192+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1a9804c6d0 con 0x7f1aa81017d0 2024-01-19T06:20:26.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.192+0000 7f1aa659c700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 0x7f1a90038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:26.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.192+0000 7f1aa659c700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 0x7f1a90038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009990 tx=0x7f1a9c006e40 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:26.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.197+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1a98011780 con 0x7f1aa81017d0 2024-01-19T06:20:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.471+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1aa810c100 con 0x7f1aa81017d0 2024-01-19T06:20:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.471+0000 7f1a8f7fe700 1 -- 172.21.15.150:0/2875684110 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1a98026360 con 0x7f1aa81017d0 2024-01-19T06:20:26.471 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:26.471 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:17:59.510503Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 msgr2=0x7f1a90038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 0x7f1a90038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009990 tx=0x7f1a9c006e40 comp rx=0 tx=0).stop 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 msgr2=0x7f1aa8111a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1a98009330 tx=0x7f1a98009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 shutdown_connections 2024-01-19T06:20:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1a900367c0 0x7f1a90038c70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 --2- 172.21.15.150:0/2875684110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa81017d0 0x7f1aa8111a70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 >> 172.21.15.150:0/2875684110 conn(0x7f1aa80f7310 msgr2=0x7f1aa80f92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 shutdown_connections 2024-01-19T06:20:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:26.475+0000 7f1aad154700 1 -- 172.21.15.150:0/2875684110 wait complete. 2024-01-19T06:20:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-01-19T06:20:27.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:26 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2875684110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:27.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:26 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:28.366 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:27 smithi134 ceph-mon[28283]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:29.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:29 smithi150 systemd[1]: Started Ceph mon.smithi150 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T06:20:29.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T06:20:29.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T06:20:29.988 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:20:34.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.556+0000 7f09b5bd2700 1 -- 172.21.15.150:0/1179033933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0998003580 msgr2=0x7f0998005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:34.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.556+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/1179033933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0998003580 0x7f0998005a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09b00fc9c0 tx=0x7f09a000b040 comp rx=0 tx=0).stop 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.556+0000 7f09b5bd2700 1 -- 172.21.15.150:0/1179033933 shutdown_connections 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.556+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/1179033933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0998003580 0x7f0998005a00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.556+0000 7f09b5bd2700 1 -- 172.21.15.150:0/1179033933 >> 172.21.15.150:0/1179033933 conn(0x7f09b00f8630 msgr2=0x7f09b00faa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.557+0000 7f09b5bd2700 1 -- 172.21.15.150:0/1179033933 shutdown_connections 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.557+0000 7f09b5bd2700 1 -- 172.21.15.150:0/1179033933 wait complete. 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.557+0000 7f09b5bd2700 1 Processor -- start 2024-01-19T06:20:34.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.557+0000 7f09b5bd2700 1 -- start start 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: mon.smithi134 calling monitor election 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: mon.smithi150 calling monitor election 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.xkxrco/crt"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:34.558 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: mon.smithi134 is new leader, mons smithi134,smithi150 in quorum (ranks 0,1) 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: monmap e2: 2 mons at {smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: fsmap 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: mgrmap e17: smithi134.qrxokr(active, since 27s) 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: Standby manager daemon smithi150.xkxrco started 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: overall HEALTH_OK 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.xkxrco/key"}]: dispatch 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.559 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.560 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09b5bd2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:34.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09b5bd2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 0x7f09b00708f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09b5bd2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0119010 con 0x7f09b0070480 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09b5bd2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0119150 con 0x7f0998003580 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33694/0 (socket says 172.21.15.150:33694) 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 -- 172.21.15.150:0/622148107 learned_addr learned my addr 172.21.15.150:0/622148107 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09aeffd700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 0x7f09b00708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 -- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 msgr2=0x7f09b006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 -- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 msgr2=0x7f09b006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09af7fe700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09aeffd700 1 -- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 msgr2=0x7f09b006ff40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09aeffd700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.558+0000 7f09aeffd700 1 -- 172.21.15.150:0/622148107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0005040 con 0x7f09b0070480 2024-01-19T06:20:34.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f09aeffd700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 0x7f09b00708f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b790 tx=0x7f09a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a4011810 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09a4011e20 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b01193b0 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09a400f2e0 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.559+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b01198c0 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.560+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b005f270 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.560+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7f09a4011970 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.561+0000 7f0997fff700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 0x7f099c06b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.561+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f09a4085fd0 con 0x7f09b0070480 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.561+0000 7f09af7fe700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 0x7f099c06b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:34.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.561+0000 7f09af7fe700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 0x7f099c06b490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f09a0000ea0 tx=0x7f09a0018040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:34.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.565+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f09a4053500 con 0x7f09b0070480 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: mon.smithi134 calling monitor election 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: mon.smithi150 calling monitor election 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.xkxrco/crt"}]: dispatch 2024-01-19T06:20:34.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: mon.smithi134 is new leader, mons smithi134,smithi150 in quorum (ranks 0,1) 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: monmap e2: 2 mons at {smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: fsmap 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: mgrmap e17: smithi134.qrxokr(active, since 27s) 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: Standby manager daemon smithi150.xkxrco started 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: overall HEALTH_OK 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.xkxrco/key"}]: dispatch 2024-01-19T06:20:34.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.? 172.21.15.150:0/530312035' entity='mgr.smithi150.xkxrco' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-19T06:20:34.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:34.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.842+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b0104680 con 0x7f09b0070480 2024-01-19T06:20:34.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.843+0000 7f0997fff700 1 -- 172.21.15.150:0/622148107 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f09a4015090 con 0x7f09b0070480 2024-01-19T06:20:34.843 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-01-19T06:20:34.843 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","modified":"2024-01-19T06:20:29.213442Z","created":"2024-01-19T06:17:59.510503Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi150","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-19T06:20:34.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.845+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 msgr2=0x7f099c06b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:34.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.845+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 0x7f099c06b490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f09a0000ea0 tx=0x7f09a0018040 comp rx=0 tx=0).stop 2024-01-19T06:20:34.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.845+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 msgr2=0x7f09b00708f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:34.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.845+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 0x7f09b00708f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f09a400b790 tx=0x7f09a40091a0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 shutdown_connections 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f099c068fe0 0x7f099c06b490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0998003580 0x7f09b006ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 --2- 172.21.15.150:0/622148107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09b0070480 0x7f09b00708f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 >> 172.21.15.150:0/622148107 conn(0x7f09b00f8630 msgr2=0x7f09b00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 shutdown_connections 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:20:34.846+0000 7f09b5bd2700 1 -- 172.21.15.150:0/622148107 wait complete. 2024-01-19T06:20:34.846 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2024-01-19T06:20:35.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: mgrmap e18: smithi134.qrxokr(active, since 27s), standbys: smithi150.xkxrco 2024-01-19T06:20:35.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi150.xkxrco", "id": "smithi150.xkxrco"}]: dispatch 2024-01-19T06:20:35.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:20:35.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:35.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/622148107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:35.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: mgrmap e18: smithi134.qrxokr(active, since 27s), standbys: smithi150.xkxrco 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr metadata", "who": "smithi150.xkxrco", "id": "smithi150.xkxrco"}]: dispatch 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/622148107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T06:20:35.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:20:35.963 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-19T06:20:35.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph config generate-minimal-conf 2024-01-19T06:20:36.228 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:20:36.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:36.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:36.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:36.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.794+0000 7fe01621b700 1 -- 172.21.15.134:0/3067915112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010071c10 msgr2=0x7fe01006ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.794+0000 7fe01621b700 1 --2- 172.21.15.134:0/3067915112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010071c10 0x7fe01006ffe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe0000048c0 tx=0x7fe000009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- 172.21.15.134:0/3067915112 shutdown_connections 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 --2- 172.21.15.134:0/3067915112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010071c10 0x7fe01006ffe0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- 172.21.15.134:0/3067915112 >> 172.21.15.134:0/3067915112 conn(0x7fe0100f8410 msgr2=0x7fe0100fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- 172.21.15.134:0/3067915112 shutdown_connections 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- 172.21.15.134:0/3067915112 wait complete. 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 Processor -- start 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- start start 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe010071c10 0x7fe0101192d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010119e20 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.795+0000 7fe01621b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0101136b0 con 0x7fe010071c10 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:52314/0 (socket says 172.21.15.134:52314) 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 -- 172.21.15.134:0/3800511880 learned_addr learned my addr 172.21.15.134:0/3800511880 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 -- 172.21.15.134:0/3800511880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe010071c10 msgr2=0x7fe0101192d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe010071c10 0x7fe0101192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 -- 172.21.15.134:0/3800511880 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe000005040 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe00effd700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe00400b790 tx=0x7fe004009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe004010870 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe004010e80 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010113990 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00400d3b0 con 0x7fe010119810 2024-01-19T06:20:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.796+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe010113ed0 con 0x7fe010119810 2024-01-19T06:20:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.797+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01005f270 con 0x7fe010119810 2024-01-19T06:20:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.798+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7fe00400f3e0 con 0x7fe010119810 2024-01-19T06:20:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.798+0000 7fdff7fff700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 0x7fdff806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.798+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe0040861f0 con 0x7fe010119810 2024-01-19T06:20:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.798+0000 7fe00f7fe700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 0x7fdff806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:37.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.798+0000 7fe00f7fe700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 0x7fdff806b4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe000000c00 tx=0x7fe000009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:37.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.802+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe004057230 con 0x7fe010119810 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: Updating smithi134:/etc/ceph/ceph.conf 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: Updating smithi150:/etc/ceph/ceph.conf 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: Updating smithi134:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: Updating smithi150:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:37.863 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T06:20:37.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.993+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe01010d090 con 0x7fe010119810 2024-01-19T06:20:37.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.993+0000 7fdff7fff700 1 -- 172.21.15.134:0/3800511880 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7fe004014100 con 0x7fe010119810 2024-01-19T06:20:37.993 INFO:teuthology.orchestra.run.smithi134.stdout:# minimal ceph.conf for 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:20:37.993 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-01-19T06:20:37.993 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 573cd99a-b692-11ee-95ae-87774f69a715 2024-01-19T06:20:37.993 INFO:teuthology.orchestra.run.smithi134.stdout: mon_host = [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.995+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 msgr2=0x7fdff806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.995+0000 7fe01621b700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 0x7fdff806b4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe000000c00 tx=0x7fe000009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.995+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 msgr2=0x7fe010113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.995+0000 7fe01621b700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe00400b790 tx=0x7fe004009250 comp rx=0 tx=0).stop 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 shutdown_connections 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fdff8069040 0x7fdff806b4f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:37.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe010071c10 0x7fe0101192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:37.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 --2- 172.21.15.134:0/3800511880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe010119810 0x7fe010113170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:37.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 >> 172.21.15.134:0/3800511880 conn(0x7fe0100f8410 msgr2=0x7fe0100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:37.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 shutdown_connections 2024-01-19T06:20:37.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:37.996+0000 7fe01621b700 1 -- 172.21.15.134:0/3800511880 wait complete. 2024-01-19T06:20:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: Updating smithi134:/etc/ceph/ceph.conf 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: Updating smithi150:/etc/ceph/ceph.conf 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: Updating smithi134:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: Updating smithi150:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T06:20:38.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:38.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:38 smithi134 ceph-mon[28283]: Reconfiguring mon.smithi134 (unknown last config time)... 2024-01-19T06:20:38.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:38 smithi134 ceph-mon[28283]: Reconfiguring daemon mon.smithi134 on smithi134 2024-01-19T06:20:38.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:38 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3800511880' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:39.001 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-19T06:20:39.001 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:20:39.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-01-19T06:20:39.032 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:20:39.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T06:20:39.099 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:20:39.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-01-19T06:20:39.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:38 smithi150 ceph-mon[35328]: Reconfiguring mon.smithi134 (unknown last config time)... 2024-01-19T06:20:39.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:38 smithi150 ceph-mon[35328]: Reconfiguring daemon mon.smithi134 on smithi134 2024-01-19T06:20:39.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:38 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3800511880' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:39.133 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:20:39.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T06:20:39.204 INFO:tasks.cephadm:Deploying OSDs... 2024-01-19T06:20:39.205 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:20:39.205 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T06:20:39.220 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T06:20:39.220 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-19 06:19:58.297324672 +0000 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-19 06:14:21.450098518 +0000 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-19 06:14:21.450098518 +0000 2024-01-19T06:20:39.275 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-19 06:14:21.450098518 +0000 2024-01-19T06:20:39.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T06:20:39.341 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-19T06:20:39.341 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-19T06:20:39.341 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000184195 s, 2.8 MB/s 2024-01-19T06:20:39.342 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T06:20:39.397 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-01-19T06:20:39.452 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-19 06:19:58.297324672 +0000 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-19 06:14:21.790090703 +0000 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-19 06:14:21.790090703 +0000 2024-01-19T06:20:39.453 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-19 06:14:21.790090703 +0000 2024-01-19T06:20:39.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T06:20:39.520 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-19T06:20:39.520 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-19T06:20:39.520 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000199307 s, 2.6 MB/s 2024-01-19T06:20:39.522 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T06:20:39.577 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-01-19T06:20:39.632 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T06:20:39.632 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:39.632 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-01-19T06:20:39.632 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:39.633 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:39.633 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-19 06:19:58.297324672 +0000 2024-01-19T06:20:39.633 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-19 06:14:22.127082960 +0000 2024-01-19T06:20:39.633 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-19 06:14:22.127082960 +0000 2024-01-19T06:20:39.633 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-19 06:14:22.127082960 +0000 2024-01-19T06:20:39.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T06:20:39.698 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-19T06:20:39.698 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-19T06:20:39.698 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00017445 s, 2.9 MB/s 2024-01-19T06:20:39.700 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T06:20:39.756 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-19 06:19:58.298324649 +0000 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-19 06:14:22.475074963 +0000 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-19 06:14:22.475074963 +0000 2024-01-19T06:20:39.811 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-19 06:14:22.475074963 +0000 2024-01-19T06:20:39.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T06:20:39.877 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-19T06:20:39.877 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-19T06:20:39.877 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000170346 s, 3.0 MB/s 2024-01-19T06:20:39.878 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T06:20:39.934 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:20:39.934 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T06:20:39.953 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T06:20:39.953 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-01-19 06:20:28.485763223 +0000 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-01-19 06:14:22.364351185 +0000 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-01-19 06:14:22.364351185 +0000 2024-01-19T06:20:40.011 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-01-19 06:14:22.364351185 +0000 2024-01-19T06:20:40.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.qrxokr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T06:20:40.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.qrxokr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T06:20:40.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:40.080 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-01-19T06:20:40.080 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-01-19T06:20:40.080 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000228141 s, 2.2 MB/s 2024-01-19T06:20:40.082 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T06:20:40.139 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-01-19T06:20:40.196 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-01-19 06:20:28.486763204 +0000 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-01-19 06:14:22.707344719 +0000 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-01-19 06:14:22.707344719 +0000 2024-01-19T06:20:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-01-19 06:14:22.707344719 +0000 2024-01-19T06:20:40.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T06:20:40.264 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-01-19T06:20:40.264 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-01-19T06:20:40.265 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.0002519 s, 2.0 MB/s 2024-01-19T06:20:40.266 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T06:20:40.322 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-01-19 06:20:28.486763204 +0000 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-01-19 06:14:23.043338385 +0000 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-01-19 06:14:23.043338385 +0000 2024-01-19T06:20:40.379 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-01-19 06:14:23.043338385 +0000 2024-01-19T06:20:40.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T06:20:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-01-19T06:20:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-01-19T06:20:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00019301 s, 2.7 MB/s 2024-01-19T06:20:40.448 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T06:20:40.504 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-01-19 06:20:28.487763185 +0000 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-01-19 06:14:23.385331938 +0000 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-01-19 06:14:23.385331938 +0000 2024-01-19T06:20:40.561 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-01-19 06:14:23.385331938 +0000 2024-01-19T06:20:40.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T06:20:40.627 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-01-19T06:20:40.627 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-01-19T06:20:40.627 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000200886 s, 2.5 MB/s 2024-01-19T06:20:40.629 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T06:20:40.685 INFO:tasks.cephadm:Deploying osd.0 on smithi134 with /dev/vg_nvme/lv_4... 2024-01-19T06:20:40.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-19T06:20:40.954 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:20:41.080 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:40 smithi134 ceph-mon[28283]: Reconfiguring mgr.smithi134.qrxokr (unknown last config time)... 2024-01-19T06:20:41.080 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:40 smithi134 ceph-mon[28283]: Reconfiguring daemon mgr.smithi134.qrxokr on smithi134 2024-01-19T06:20:41.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:40 smithi150 ceph-mon[35328]: Reconfiguring mgr.smithi134.qrxokr (unknown last config time)... 2024-01-19T06:20:41.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:40 smithi150 ceph-mon[35328]: Reconfiguring daemon mgr.smithi134.qrxokr on smithi134 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: Reconfiguring crash.smithi134 (monmap changed)... 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:41 smithi134 ceph-mon[28283]: Reconfiguring daemon crash.smithi134 on smithi134 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: Reconfiguring crash.smithi134 (monmap changed)... 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:42.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:41 smithi150 ceph-mon[35328]: Reconfiguring daemon crash.smithi134 on smithi134 2024-01-19T06:20:43.480 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-19T06:20:43.481 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-19T06:20:43.481 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-01-19T06:20:43.481 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-01-19T06:20:43.481 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112279 s, 934 MB/s 2024-01-19T06:20:43.481 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-01-19T06:20:44.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-01-19T06:20:44.197 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:43 smithi134 ceph-mon[28283]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:44.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:43 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:44.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:43 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:44.266 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:43 smithi150 ceph-mon[35328]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:44.266 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:43 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:44.266 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:43 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:44.998 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:20:45.214 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:44 smithi134 ceph-mon[28283]: Reconfiguring alertmanager.smithi134 (dependencies changed)... 2024-01-19T06:20:45.214 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:44 smithi134 ceph-mon[28283]: Reconfiguring daemon alertmanager.smithi134 on smithi134 2024-01-19T06:20:45.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:44 smithi150 ceph-mon[35328]: Reconfiguring alertmanager.smithi134 (dependencies changed)... 2024-01-19T06:20:45.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:44 smithi150 ceph-mon[35328]: Reconfiguring daemon alertmanager.smithi134 on smithi134 2024-01-19T06:20:46.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:45 smithi134 ceph-mon[28283]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:46.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:45 smithi150 ceph-mon[35328]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.858+0000 7f75f8706700 1 -- 172.21.15.134:0/3724597349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fde00 msgr2=0x7f75f00fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.858+0000 7f75f8706700 1 --2- 172.21.15.134:0/3724597349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fde00 0x7f75f00fe250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75e0004940 tx=0x7f75e0009e30 comp rx=0 tx=0).stop 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- 172.21.15.134:0/3724597349 shutdown_connections 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 --2- 172.21.15.134:0/3724597349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fde00 0x7f75f00fe250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 --2- 172.21.15.134:0/3724597349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fcbf0 0x7f75f00fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- 172.21.15.134:0/3724597349 >> 172.21.15.134:0/3724597349 conn(0x7f75f00f82d0 msgr2=0x7f75f00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- 172.21.15.134:0/3724597349 shutdown_connections 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- 172.21.15.134:0/3724597349 wait complete. 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 Processor -- start 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- start start 2024-01-19T06:20:46.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fde00 0x7f75f010daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f010e130 con 0x7f75f00fcbf0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.859+0000 7f75f8706700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f010e970 con 0x7f75f00fde00 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43710/0 (socket says 172.21.15.134:43710) 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 -- 172.21.15.134:0/1441166944 learned_addr learned my addr 172.21.15.134:0/1441166944 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f5ca1700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fde00 0x7f75f010daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 -- 172.21.15.134:0/1441166944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fde00 msgr2=0x7f75f010daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fde00 0x7f75f010daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 -- 172.21.15.134:0/1441166944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e0005040 con 0x7f75f00fcbf0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f64a2700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b790 tx=0x7f75ec009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec00f6f0 con 0x7f75f00fcbf0 2024-01-19T06:20:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75ec00fd00 con 0x7f75f00fcbf0 2024-01-19T06:20:46.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f010ec50 con 0x7f75f00fcbf0 2024-01-19T06:20:46.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec011e90 con 0x7f75f00fcbf0 2024-01-19T06:20:46.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.860+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f010f190 con 0x7f75f00fcbf0 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.861+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f004bb80 con 0x7f75f00fcbf0 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.861+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7f75ec00f850 con 0x7f75f00fcbf0 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.862+0000 7f75e6ffd700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 0x7f75dc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.862+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f75ec085a90 con 0x7f75f00fcbf0 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.862+0000 7f75f5ca1700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 0x7f75dc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:20:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.862+0000 7f75f5ca1700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 0x7f75dc06b380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75e000b3b0 tx=0x7f75e0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:20:46.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:46.866+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f75ec056ad0 con 0x7f75f00fcbf0 2024-01-19T06:20:47.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:20:47.065+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f75f010f610 con 0x7f75dc068ed0 2024-01-19T06:20:48.225 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:47 smithi134 ceph-mon[28283]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:48.226 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:47 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:20:48.226 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:47 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:20:48.226 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:47 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:47 smithi150 ceph-mon[35328]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:47 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:20:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:47 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:20:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:47 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:20:49.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:48 smithi134 ceph-mon[28283]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:20:49.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:48 smithi150 ceph-mon[35328]: from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:20:50.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:49 smithi134 ceph-mon[28283]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:50.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:49 smithi150 ceph-mon[35328]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:52.412 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:52 smithi134 ceph-mon[28283]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:52.412 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:52.413 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:52.413 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:52 smithi150 ceph-mon[35328]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:20:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:20:53.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: Reconfiguring grafana.smithi134 (dependencies changed)... 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: Reconfiguring daemon grafana.smithi134 on smithi134 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2379960320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6866d2e8-d629-4fd8-ab59-37109281459b"}]: dispatch 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2379960320' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6866d2e8-d629-4fd8-ab59-37109281459b"}]': finished 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: osdmap e6: 1 total, 0 up, 1 in 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:20:53.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:53 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1506910607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: Reconfiguring grafana.smithi134 (dependencies changed)... 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: Reconfiguring daemon grafana.smithi134 on smithi134 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2379960320' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6866d2e8-d629-4fd8-ab59-37109281459b"}]: dispatch 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2379960320' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6866d2e8-d629-4fd8-ab59-37109281459b"}]': finished 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: osdmap e6: 1 total, 0 up, 1 in 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:20:53.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:53 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1506910607' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:20:54.336 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:54 smithi150 ceph-mon[35328]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:54.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:54 smithi134 ceph-mon[28283]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:55.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:55 smithi134 ceph-mon[28283]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:55 smithi150 ceph-mon[35328]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:58.342 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:57 smithi134 ceph-mon[28283]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:20:58.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:57 smithi150 ceph-mon[35328]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:00.276 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:20:59 smithi134 ceph-mon[28283]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:00.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:20:59 smithi150 ceph-mon[35328]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:02.060 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:01 smithi134 ceph-mon[28283]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:02.060 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:01 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:02.060 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:01 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:02.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:01 smithi150 ceph-mon[35328]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:02.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:01 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:02.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:01 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:03.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:02 smithi134 ceph-mon[28283]: Reconfiguring prometheus.smithi134 (dependencies changed)... 2024-01-19T06:21:03.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:02 smithi134 ceph-mon[28283]: Reconfiguring daemon prometheus.smithi134 on smithi134 2024-01-19T06:21:03.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-19T06:21:03.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:02 smithi150 ceph-mon[35328]: Reconfiguring prometheus.smithi134 (dependencies changed)... 2024-01-19T06:21:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:02 smithi150 ceph-mon[35328]: Reconfiguring daemon prometheus.smithi134 on smithi134 2024-01-19T06:21:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-19T06:21:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:04.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:03 smithi134 ceph-mon[28283]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:04.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:03 smithi134 ceph-mon[28283]: Deploying daemon osd.0 on smithi134 2024-01-19T06:21:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:03 smithi150 ceph-mon[35328]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:03 smithi150 ceph-mon[35328]: Deploying daemon osd.0 on smithi134 2024-01-19T06:21:06.114 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:05 smithi134 ceph-mon[28283]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:06.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:05 smithi150 ceph-mon[35328]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:07.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:07.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:07.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:07.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:21:07.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:07.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:07.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:07.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:07.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T06:21:07.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:08.278 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:07 smithi134 ceph-mon[28283]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:08.278 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:07 smithi134 ceph-mon[28283]: Reconfiguring crash.smithi150 (monmap changed)... 2024-01-19T06:21:08.278 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:07 smithi134 ceph-mon[28283]: Reconfiguring daemon crash.smithi150 on smithi150 2024-01-19T06:21:08.381 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:07 smithi150 ceph-mon[35328]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:08.382 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:07 smithi150 ceph-mon[35328]: Reconfiguring crash.smithi150 (monmap changed)... 2024-01-19T06:21:08.382 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:07 smithi150 ceph-mon[35328]: Reconfiguring daemon crash.smithi150 on smithi150 2024-01-19T06:21:09.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:09.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:09.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.xkxrco", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T06:21:09.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T06:21:09.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:09 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:09.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:09 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:09.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:09 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:09.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:09 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.xkxrco", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T06:21:09.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:09 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T06:21:09.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:09 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: Reconfiguring mgr.smithi150.xkxrco (monmap changed)... 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: Reconfiguring daemon mgr.smithi150.xkxrco on smithi150 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T06:21:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:10.367 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: Reconfiguring mgr.smithi150.xkxrco (monmap changed)... 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: Reconfiguring daemon mgr.smithi150.xkxrco on smithi150 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T06:21:10.368 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:11.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: Reconfiguring mon.smithi150 (monmap changed)... 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: Reconfiguring daemon mon.smithi150 on smithi150 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: Reconfiguring mon.smithi150 (monmap changed)... 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: Reconfiguring daemon mon.smithi150 on smithi150 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T06:21:11.416 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T06:21:11.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T06:21:11.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:11.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T06:21:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:12 smithi150 ceph-mon[35328]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi134.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T06:21:12.284 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:12 smithi134 ceph-mon[28283]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi134.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T06:21:13.117 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.117 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.118 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.118 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:13 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:13 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:13 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:13 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:13.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:13 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:14.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:14 smithi134 ceph-mon[28283]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:14.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:14 smithi150 ceph-mon[35328]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:15.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:15 smithi134 ceph-mon[28283]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:15.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:15 smithi150 ceph-mon[35328]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:15.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.897+0000 7f75e6ffd700 1 -- 172.21.15.134:0/1441166944 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75f010f610 con 0x7f75dc068ed0 2024-01-19T06:21:15.897 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 0 on host 'smithi134' 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 msgr2=0x7f75dc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 0x7f75dc06b380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75e000b3b0 tx=0x7f75e0009e30 comp rx=0 tx=0).stop 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 msgr2=0x7f75f010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b790 tx=0x7f75ec009220 comp rx=0 tx=0).stop 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 shutdown_connections 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f75dc068ed0 0x7f75dc06b380 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75f00fcbf0 0x7f75f010d560 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 --2- 172.21.15.134:0/1441166944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f00fde00 0x7f75f010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 >> 172.21.15.134:0/1441166944 conn(0x7f75f00f82d0 msgr2=0x7f75f0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:21:15.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 shutdown_connections 2024-01-19T06:21:15.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:15.899+0000 7f75f8706700 1 -- 172.21.15.134:0/1441166944 wait complete. 2024-01-19T06:21:16.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:16.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:16.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:16 smithi134 ceph-mon[28283]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-19T06:21:16.933 DEBUG:teuthology.orchestra.run.smithi134:osd.0> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.0.service 2024-01-19T06:21:16.976 INFO:tasks.cephadm:Deploying osd.1 on smithi134 with /dev/vg_nvme/lv_3... 2024-01-19T06:21:16.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-19T06:21:17.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:17.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:17.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:16 smithi150 ceph-mon[35328]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-19T06:21:17.594 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:21:17.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-mon[28283]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:17.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-mon[28283]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-19T06:21:17.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-mon[28283]: osdmap e7: 1 total, 0 up, 1 in 2024-01-19T06:21:17.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:17.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-mon[28283]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:21:17.975 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:21:17.879+0000 7fecd917e700 -1 osd.0 0 waiting for initial osdmap 2024-01-19T06:21:17.975 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:21:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:21:17.888+0000 7fecd577c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:21:18.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:17 smithi150 ceph-mon[35328]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:18.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:17 smithi150 ceph-mon[35328]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-19T06:21:18.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:17 smithi150 ceph-mon[35328]: osdmap e7: 1 total, 0 up, 1 in 2024-01-19T06:21:18.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:18.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:17 smithi150 ceph-mon[35328]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:21:19.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:18 smithi134 ceph-mon[28283]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:21:19.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:18 smithi134 ceph-mon[28283]: osdmap e8: 1 total, 0 up, 1 in 2024-01-19T06:21:19.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:18 smithi150 ceph-mon[35328]: from='osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:21:19.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:18 smithi150 ceph-mon[35328]: osdmap e8: 1 total, 0 up, 1 in 2024-01-19T06:21:19.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] boot 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: osdmap e9: 1 total, 1 up, 1 in 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:19.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:19 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:19.980 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-19T06:21:19.981 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-19T06:21:19.981 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-01-19T06:21:19.981 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-01-19T06:21:19.981 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130287 s, 805 MB/s 2024-01-19T06:21:19.981 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: osd.0 [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] boot 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: osdmap e9: 1 total, 1 up, 1 in 2024-01-19T06:21:20.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T06:21:20.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:20.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:19 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:20.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-01-19T06:21:21.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:20 smithi134 ceph-mon[28283]: osdmap e10: 1 total, 1 up, 1 in 2024-01-19T06:21:21.181 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:20 smithi150 ceph-mon[35328]: osdmap e10: 1 total, 1 up, 1 in 2024-01-19T06:21:22.059 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:21 smithi134 ceph-mon[28283]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:22.059 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:21 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:22.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:21 smithi150 ceph-mon[35328]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:22.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:21 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:22.377 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:24.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:24.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:24.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:24.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 -- 172.21.15.134:0/2347058275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fcbe0 msgr2=0x7fb9680fcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:24.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 --2- 172.21.15.134:0/2347058275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fcbe0 0x7fb9680fcff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb958004940 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 -- 172.21.15.134:0/2347058275 shutdown_connections 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 --2- 172.21.15.134:0/2347058275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fde90 0x7fb9680fe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 --2- 172.21.15.134:0/2347058275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fcbe0 0x7fb9680fcff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 -- 172.21.15.134:0/2347058275 >> 172.21.15.134:0/2347058275 conn(0x7fb9680f8360 msgr2=0x7fb9680fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 -- 172.21.15.134:0/2347058275 shutdown_connections 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 -- 172.21.15.134:0/2347058275 wait complete. 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.611+0000 7fb96e01b700 1 Processor -- start 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb96e01b700 1 -- start start 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb96e01b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fcbe0 0x7fb96810d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb96e01b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb96e01b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96810e010 con 0x7fb9680fde90 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb96e01b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96810e8b0 con 0x7fb9680fcbe0 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:24.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36152/0 (socket says 172.21.15.134:36152) 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 -- 172.21.15.134:0/3346342099 learned_addr learned my addr 172.21.15.134:0/3346342099 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb9677fe700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fcbe0 0x7fb96810d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 -- 172.21.15.134:0/3346342099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fcbe0 msgr2=0x7fb96810d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fcbe0 0x7fb96810d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 -- 172.21.15.134:0/3346342099 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958005040 con 0x7fb9680fde90 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb966ffd700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb95c002a80 tx=0x7fb95c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.612+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95c007cc0 con 0x7fb9680fde90 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.613+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96810eb90 con 0x7fb9680fde90 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.613+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb95c00f040 con 0x7fb9680fde90 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.613+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95c00da60 con 0x7fb9680fde90 2024-01-19T06:21:24.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.613+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96810f0d0 con 0x7fb9680fde90 2024-01-19T06:21:24.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.614+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96804bb80 con 0x7fb9680fde90 2024-01-19T06:21:24.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.614+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7fb95c00dbc0 con 0x7fb9680fde90 2024-01-19T06:21:24.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.614+0000 7fb94ffff700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 0x7fb95006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:24.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.615+0000 7fb9677fe700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 0x7fb95006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:24.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.615+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fb95c085b50 con 0x7fb9680fde90 2024-01-19T06:21:24.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.615+0000 7fb9677fe700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 0x7fb95006b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b3b0 tx=0x7fb958009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:21:24.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.619+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb95c0568b0 con 0x7fb9680fde90 2024-01-19T06:21:24.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:24.822+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb96810f520 con 0x7fb950068ed0 2024-01-19T06:21:25.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:24 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi134 2024-01-19T06:21:25.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:24 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi134 to 12864M 2024-01-19T06:21:25.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:24 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:21:25.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:24 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:21:25.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:24 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:25.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:24 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi134 2024-01-19T06:21:25.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:24 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi134 to 12864M 2024-01-19T06:21:25.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:24 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:21:25.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:24 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:21:25.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:24 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:26.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:25 smithi134 ceph-mon[28283]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:26.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:25 smithi134 ceph-mon[28283]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:21:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:25 smithi150 ceph-mon[35328]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:25 smithi150 ceph-mon[35328]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:21:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:28.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:27 smithi134 ceph-mon[28283]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:28.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:27 smithi150 ceph-mon[35328]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:29.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:28 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2828196684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8d35ff7-48c7-49f5-afa6-e43f5d10212f"}]: dispatch 2024-01-19T06:21:29.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:28 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2828196684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8d35ff7-48c7-49f5-afa6-e43f5d10212f"}]': finished 2024-01-19T06:21:29.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:28 smithi134 ceph-mon[28283]: osdmap e11: 2 total, 1 up, 2 in 2024-01-19T06:21:29.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:28 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:29.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:28 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3284312865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:21:29.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:28 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2828196684' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8d35ff7-48c7-49f5-afa6-e43f5d10212f"}]: dispatch 2024-01-19T06:21:29.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:28 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2828196684' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8d35ff7-48c7-49f5-afa6-e43f5d10212f"}]': finished 2024-01-19T06:21:29.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:28 smithi150 ceph-mon[35328]: osdmap e11: 2 total, 1 up, 2 in 2024-01-19T06:21:29.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:28 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:29.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:28 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3284312865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:21:30.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:29 smithi134 ceph-mon[28283]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:30.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:29 smithi150 ceph-mon[35328]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:31.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:31.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:31.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:31 smithi134 ceph-mon[28283]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:31.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:31.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:31.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:31.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:31 smithi150 ceph-mon[35328]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:34.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:33 smithi134 ceph-mon[28283]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:33 smithi150 ceph-mon[35328]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:36.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:35 smithi134 ceph-mon[28283]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:36.067 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-19T06:21:36.067 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:35 smithi150 ceph-mon[35328]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-19T06:21:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:37.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:36 smithi134 ceph-mon[28283]: Deploying daemon osd.1 on smithi134 2024-01-19T06:21:37.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:36 smithi150 ceph-mon[35328]: Deploying daemon osd.1 on smithi134 2024-01-19T06:21:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:38.308 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:37 smithi134 ceph-mon[28283]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:38.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:37 smithi150 ceph-mon[35328]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:40.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:39 smithi134 ceph-mon[28283]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:40.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:39 smithi150 ceph-mon[35328]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:41.871 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:41.871 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:41.871 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:41 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:41.871 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:41 smithi134 ceph-mon[28283]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:41.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:41.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:41.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:41 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:41.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:41 smithi150 ceph-mon[35328]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:43.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:43 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:43.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:43 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:43.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:43 smithi134 ceph-mon[28283]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:43.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:43 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:43.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:43 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:43.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:43 smithi150 ceph-mon[35328]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.684+0000 7fb94ffff700 1 -- 172.21.15.134:0/3346342099 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb96810f520 con 0x7fb950068ed0 2024-01-19T06:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 1 on host 'smithi134' 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 msgr2=0x7fb95006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 0x7fb95006b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb95800b3b0 tx=0x7fb958009e30 comp rx=0 tx=0).stop 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 msgr2=0x7fb96810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb95c002a80 tx=0x7fb95c0046b0 comp rx=0 tx=0).stop 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 shutdown_connections 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb950068ed0 0x7fb95006b380 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9680fcbe0 0x7fb96810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 --2- 172.21.15.134:0/3346342099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9680fde90 0x7fb96810da10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 >> 172.21.15.134:0/3346342099 conn(0x7fb9680f8360 msgr2=0x7fb9681010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 shutdown_connections 2024-01-19T06:21:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:43.686+0000 7fb96e01b700 1 -- 172.21.15.134:0/3346342099 wait complete. 2024-01-19T06:21:44.295 DEBUG:teuthology.orchestra.run.smithi134:osd.1> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.1.service 2024-01-19T06:21:44.297 INFO:tasks.cephadm:Deploying osd.2 on smithi134 with /dev/vg_nvme/lv_2... 2024-01-19T06:21:44.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-19T06:21:44.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:21:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:21:44.387+0000 7fc89dc55540 -1 osd.1 0 log_to_monitors true 2024-01-19T06:21:44.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:44 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:44.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:44 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:44.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:44 smithi134 ceph-mon[28283]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-19T06:21:44.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:44 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:44.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:44 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:44.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:44 smithi150 ceph-mon[35328]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-19T06:21:44.955 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:21:45.923 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:21:45.652+0000 7fc894429700 -1 osd.1 0 waiting for initial osdmap 2024-01-19T06:21:45.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:21:45.658+0000 7fc890226700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:21:45.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-mon[28283]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:45.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-mon[28283]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-19T06:21:45.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-mon[28283]: osdmap e12: 2 total, 1 up, 2 in 2024-01-19T06:21:45.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:45.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:45 smithi134 ceph-mon[28283]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:21:45.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:45 smithi150 ceph-mon[35328]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:45.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:45 smithi150 ceph-mon[35328]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-19T06:21:45.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:45 smithi150 ceph-mon[35328]: osdmap e12: 2 total, 1 up, 2 in 2024-01-19T06:21:45.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:45 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:45.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:45 smithi150 ceph-mon[35328]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:21:46.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:46 smithi134 ceph-mon[28283]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:21:46.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:46 smithi134 ceph-mon[28283]: osdmap e13: 2 total, 1 up, 2 in 2024-01-19T06:21:46.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:46.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:46.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:46 smithi150 ceph-mon[35328]: from='osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:21:46.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:46 smithi150 ceph-mon[35328]: osdmap e13: 2 total, 1 up, 2 in 2024-01-19T06:21:46.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:46.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129041 s, 813 MB/s 2024-01-19T06:21:47.442 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] boot 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: osdmap e14: 2 total, 2 up, 2 in 2024-01-19T06:21:47.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:47 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: osd.1 [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] boot 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: osdmap e14: 2 total, 2 up, 2 in 2024-01-19T06:21:47.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:47 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T06:21:48.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-01-19T06:21:49.008 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:21:49.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:48 smithi134 ceph-mon[28283]: osdmap e15: 2 total, 2 up, 2 in 2024-01-19T06:21:49.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:48 smithi150 ceph-mon[35328]: osdmap e15: 2 total, 2 up, 2 in 2024-01-19T06:21:50.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:49 smithi150 ceph-mon[35328]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:50.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:49 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:50.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:49 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:50.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:49 smithi134 ceph-mon[28283]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:50.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:49 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:50.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:49 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:50.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 -- 172.21.15.134:0/2144826796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fcd00 msgr2=0x7f7f380fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/2144826796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fcd00 0x7f7f380fd150 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7f28004940 tx=0x7f7f28009e30 comp rx=0 tx=0).stop 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 -- 172.21.15.134:0/2144826796 shutdown_connections 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/2144826796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fcd00 0x7f7f380fd150 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/2144826796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fbaf0 0x7f7f380fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 -- 172.21.15.134:0/2144826796 >> 172.21.15.134:0/2144826796 conn(0x7f7f380f71d0 msgr2=0x7f7f380f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.536+0000 7f7f3fe32700 1 -- 172.21.15.134:0/2144826796 shutdown_connections 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 -- 172.21.15.134:0/2144826796 wait complete. 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 Processor -- start 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 -- start start 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fcd00 0x7f7f3810c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3810d010 con 0x7f7f380fbaf0 2024-01-19T06:21:50.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3fe32700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3810d850 con 0x7f7f380fcd00 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3dbce700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3dbce700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34334/0 (socket says 172.21.15.134:34334) 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.537+0000 7f7f3dbce700 1 -- 172.21.15.134:0/1429464543 learned_addr learned my addr 172.21.15.134:0/1429464543 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3d3cd700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fcd00 0x7f7f3810c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3dbce700 1 -- 172.21.15.134:0/1429464543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fcd00 msgr2=0x7f7f3810c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3dbce700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fcd00 0x7f7f3810c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3dbce700 1 -- 172.21.15.134:0/1429464543 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28005040 con 0x7f7f380fbaf0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3dbce700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400b790 tx=0x7f7f34009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3400f6f0 con 0x7f7f380fbaf0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f3400fd00 con 0x7f7f380fbaf0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3810db30 con 0x7f7f380fbaf0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f340103a0 con 0x7f7f380fbaf0 2024-01-19T06:21:50.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.538+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3810e070 con 0x7f7f380fbaf0 2024-01-19T06:21:50.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.539+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3804bb80 con 0x7f7f380fbaf0 2024-01-19T06:21:50.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.539+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7f7f3400f850 con 0x7f7f380fbaf0 2024-01-19T06:21:50.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.540+0000 7f7f2e7fc700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 0x7f7f2406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:21:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.540+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f7f34085ed0 con 0x7f7f380fbaf0 2024-01-19T06:21:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.540+0000 7f7f3d3cd700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 0x7f7f2406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:21:50.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.540+0000 7f7f3d3cd700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 0x7f7f2406b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f28004560 tx=0x7f7f28009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:21:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.544+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7f34056a80 con 0x7f7f380fbaf0 2024-01-19T06:21:50.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:21:50.739+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f3810e520 con 0x7f7f24068ed0 2024-01-19T06:21:51.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:21:51.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:21:51.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:51.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:21:51.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:21:51.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:52.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:51 smithi134 ceph-mon[28283]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:52.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:51 smithi134 ceph-mon[28283]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:21:52.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:51 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:51 smithi150 ceph-mon[35328]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:51 smithi150 ceph-mon[35328]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:21:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:51 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:21:54.303 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:53 smithi134 ceph-mon[28283]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:54.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:53 smithi150 ceph-mon[35328]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi134 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi134 to 6432M 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:55.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:55 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi134 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:21:55.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi134 to 6432M 2024-01-19T06:21:55.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:55.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:55.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:55.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:55 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3522905362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2df8d4e-2a3c-479a-8033-2b777cf50888"}]: dispatch 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3522905362' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2df8d4e-2a3c-479a-8033-2b777cf50888"}]': finished 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: osdmap e16: 3 total, 2 up, 3 in 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:21:56.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:56 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/662928214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3522905362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2df8d4e-2a3c-479a-8033-2b777cf50888"}]: dispatch 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3522905362' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2df8d4e-2a3c-479a-8033-2b777cf50888"}]': finished 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: osdmap e16: 3 total, 2 up, 3 in 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:21:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:56 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/662928214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:21:58.223 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:57 smithi134 ceph-mon[28283]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:58.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:57 smithi150 ceph-mon[35328]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:59.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:59.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:59.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:59.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:59.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:59.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:21:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:21:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:21:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:21:59.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:21:59.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:21:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:01.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:01 smithi134 ceph-mon[28283]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:01.931 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:01 smithi150 ceph-mon[35328]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:03.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-19T06:22:03.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-19T06:22:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:04.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:03 smithi134 ceph-mon[28283]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:04.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:03 smithi134 ceph-mon[28283]: Deploying daemon osd.2 on smithi134 2024-01-19T06:22:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:03 smithi150 ceph-mon[35328]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:03 smithi150 ceph-mon[35328]: Deploying daemon osd.2 on smithi134 2024-01-19T06:22:06.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:05 smithi134 ceph-mon[28283]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:06.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:05 smithi150 ceph-mon[35328]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:07.083 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:07.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:08.276 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:07 smithi134 ceph-mon[28283]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:08.276 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:08.276 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:08.276 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:07 smithi150 ceph-mon[35328]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:07 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:07 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:07 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:10.032 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:09 smithi134 ceph-mon[28283]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:10.435 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:09 smithi150 ceph-mon[35328]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:10.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.894+0000 7f7f2e7fc700 1 -- 172.21.15.134:0/1429464543 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7f3810e520 con 0x7f7f24068ed0 2024-01-19T06:22:10.895 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 2 on host 'smithi134' 2024-01-19T06:22:10.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 msgr2=0x7f7f2406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:10.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 0x7f7f2406b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7f28004560 tx=0x7f7f28009e30 comp rx=0 tx=0).stop 2024-01-19T06:22:10.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 msgr2=0x7f7f3810c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:10.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7f3400b790 tx=0x7f7f34009220 comp rx=0 tx=0).stop 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 shutdown_connections 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f24068ed0 0x7f7f2406b380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f380fbaf0 0x7f7f3810c440 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 --2- 172.21.15.134:0/1429464543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f380fcd00 0x7f7f3810c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 >> 172.21.15.134:0/1429464543 conn(0x7f7f380f71d0 msgr2=0x7f7f380fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 shutdown_connections 2024-01-19T06:22:10.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:22:10.897+0000 7f7f3fe32700 1 -- 172.21.15.134:0/1429464543 wait complete. 2024-01-19T06:22:11.582 DEBUG:teuthology.orchestra.run.smithi134:osd.2> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.2.service 2024-01-19T06:22:11.584 INFO:tasks.cephadm:Deploying osd.3 on smithi150 with /dev/vg_nvme/lv_4... 2024-01-19T06:22:11.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-19T06:22:11.839 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:22:11.617+0000 7f9758b36540 -1 osd.2 0 log_to_monitors true 2024-01-19T06:22:11.843 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:22:11.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:11 smithi150 ceph-mon[35328]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:11.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:11.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:11.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:11 smithi150 ceph-mon[35328]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-19T06:22:12.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:11 smithi134 ceph-mon[28283]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:12.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:12.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:12.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:11 smithi134 ceph-mon[28283]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-19T06:22:13.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:12 smithi150 ceph-mon[35328]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-19T06:22:13.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:12 smithi150 ceph-mon[35328]: osdmap e17: 3 total, 2 up, 3 in 2024-01-19T06:22:13.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:12 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:13.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:12 smithi150 ceph-mon[35328]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:22:13.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:12 smithi134 ceph-mon[28283]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-19T06:22:13.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:12 smithi134 ceph-mon[28283]: osdmap e17: 3 total, 2 up, 3 in 2024-01-19T06:22:13.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:12 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:13.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:12 smithi134 ceph-mon[28283]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-01-19T06:22:13.295 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:22:12.995+0000 7f974f30a700 -1 osd.2 0 waiting for initial osdmap 2024-01-19T06:22:13.295 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:22:13.002+0000 7f9749103700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:22:13.620 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-19T06:22:13.620 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-19T06:22:13.620 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-01-19T06:22:13.620 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-01-19T06:22:13.620 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113471 s, 924 MB/s 2024-01-19T06:22:13.621 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-01-19T06:22:14.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:14 smithi134 ceph-mon[28283]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:14 smithi134 ceph-mon[28283]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:22:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:14 smithi134 ceph-mon[28283]: osdmap e18: 3 total, 2 up, 3 in 2024-01-19T06:22:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:14 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:14.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-01-19T06:22:14.230 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:14 smithi150 ceph-mon[35328]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:14.231 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:14 smithi150 ceph-mon[35328]: from='osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-01-19T06:22:14.231 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:14 smithi150 ceph-mon[35328]: osdmap e18: 3 total, 2 up, 3 in 2024-01-19T06:22:14.231 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:14 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:14.231 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:14 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:22:15.127 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] boot 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: osdmap e19: 3 total, 3 up, 3 in 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:15.128 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:15 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: osd.2 [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] boot 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: osdmap e19: 3 total, 3 up, 3 in 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:15.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:15 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-19T06:22:15.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 -- 172.21.15.150:0/510896355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 msgr2=0x7f0d900fb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 --2- 172.21.15.150:0/510896355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fb260 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0d80004940 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 -- 172.21.15.150:0/510896355 shutdown_connections 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 --2- 172.21.15.150:0/510896355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 0x7f0d900fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 --2- 172.21.15.150:0/510896355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fb260 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 -- 172.21.15.150:0/510896355 >> 172.21.15.150:0/510896355 conn(0x7f0d900f68b0 msgr2=0x7f0d900f8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 -- 172.21.15.150:0/510896355 shutdown_connections 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.913+0000 7f0d964ba700 1 -- 172.21.15.150:0/510896355 wait complete. 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 Processor -- start 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 -- start start 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:15.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 0x7f0d900fbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900fc3f0 con 0x7f0d900fdee0 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d964ba700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900fc560 con 0x7f0d900fae50 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53200/0 (socket says 172.21.15.150:53200) 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 -- 172.21.15.150:0/3999455837 learned_addr learned my addr 172.21.15.150:0/3999455837 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8f7fe700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 0x7f0d900fbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 -- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 msgr2=0x7f0d900fbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 0x7f0d900fbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.914+0000 7f0d8ffff700 1 -- 172.21.15.150:0/3999455837 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80005040 con 0x7f0d900fae50 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d8ffff700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0d80000c00 tx=0x7f0d80009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8000e040 con 0x7f0d900fae50 2024-01-19T06:22:15.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d80009af0 con 0x7f0d900fae50 2024-01-19T06:22:15.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d900fc7e0 con 0x7f0d900fae50 2024-01-19T06:22:15.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8001c630 con 0x7f0d900fae50 2024-01-19T06:22:15.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.915+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d90115440 con 0x7f0d900fae50 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.916+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9004bb80 con 0x7f0d900fae50 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.916+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 86393+0+0 (secure 0 0 0) 0x7f0d8000bc50 con 0x7f0d900fae50 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.917+0000 7f0d8cff9700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 0x7f0d7806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.917+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f0d80087720 con 0x7f0d900fae50 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.917+0000 7f0d8f7fe700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 0x7f0d7806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:15.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.917+0000 7f0d8f7fe700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 0x7f0d7806b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009fd0 tx=0x7f0d8400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:22:15.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:15.921+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0d80058020 con 0x7f0d900fae50 2024-01-19T06:22:16.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:16.116+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d90102830 con 0x7f0d78068ed0 2024-01-19T06:22:16.157 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:16 smithi134 ceph-mon[28283]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:16.157 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-19T06:22:16.157 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:16 smithi134 ceph-mon[28283]: osdmap e20: 3 total, 3 up, 3 in 2024-01-19T06:22:16.157 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-19T06:22:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:16 smithi150 ceph-mon[35328]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T06:22:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-19T06:22:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:16 smithi150 ceph-mon[35328]: osdmap e20: 3 total, 3 up, 3 in 2024-01-19T06:22:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-19T06:22:17.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:22:17.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:22:17.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:17.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-19T06:22:17.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:17 smithi150 ceph-mon[35328]: osdmap e21: 3 total, 3 up, 3 in 2024-01-19T06:22:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:22:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:22:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-19T06:22:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:17 smithi134 ceph-mon[28283]: osdmap e21: 3 total, 3 up, 3 in 2024-01-19T06:22:17.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66718]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T06:22:17.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:17.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66718]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:17.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66718]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:17.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66721]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T06:22:17.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:17.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66721]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:17.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66721]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:18.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:18.017+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f0d80054960 con 0x7f0d900fae50 2024-01-19T06:22:18.043 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66756]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T06:22:18.043 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:18.043 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66756]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:18.043 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:17 smithi134 sudo[66756]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: osdmap e22: 3 total, 3 up, 3 in 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:18.297 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66853]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T06:22:18.297 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:18.297 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66853]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:18.297 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66853]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: osdmap e22: 3 total, 3 up, 3 in 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:18.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:18 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:18.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66916]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T06:22:18.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66916]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:18.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66916]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:18.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66916]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:18.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66938]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T06:22:18.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66938]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:18.816 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66938]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:18.816 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66938]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:19.091 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66964]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T06:22:19.091 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:19.091 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66964]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:19.091 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:18 smithi134 sudo[66964]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 sudo[67025]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 sudo[67025]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 sudo[67025]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 sudo[67025]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi134 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi134 to 4288M 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: mgrmap e19: smithi134.qrxokr(active, since 2m), standbys: smithi150.xkxrco 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/258097668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]: dispatch 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]: dispatch 2024-01-19T06:22:19.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]': finished 2024-01-19T06:22:19.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: osdmap e23: 4 total, 3 up, 4 in 2024-01-19T06:22:19.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:19.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:19 smithi134 ceph-mon[28283]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T06:22:19.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi134 2024-01-19T06:22:19.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi134 to 4288M 2024-01-19T06:22:19.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: mgrmap e19: smithi134.qrxokr(active, since 2m), standbys: smithi150.xkxrco 2024-01-19T06:22:19.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/258097668' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]: dispatch 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]: dispatch 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b66525f9-4493-4708-bbf7-e76de91fd5b5"}]': finished 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: osdmap e23: 4 total, 3 up, 4 in 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 ceph-mon[35328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38836]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38836]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:19.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38836]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:19.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38836]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:19.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38862]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T06:22:19.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38862]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T06:22:19.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38862]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T06:22:19.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:19 smithi150 sudo[38862]: pam_unix(sudo:session): session closed for user root 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/1335089349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T06:22:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:20 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/1335089349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:22:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T06:22:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T06:22:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:20 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:21.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:21 smithi134 ceph-mon[28283]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:21.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:21 smithi150 ceph-mon[35328]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:22.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:23.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:23.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:23.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:23 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:23.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:23 smithi150 ceph-mon[35328]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:23.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:23.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:23.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:23 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:23.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:23 smithi134 ceph-mon[28283]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:25.776 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:25 smithi150 ceph-mon[35328]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s 2024-01-19T06:22:25.776 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-19T06:22:25.776 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:25.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:25 smithi134 ceph-mon[28283]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s 2024-01-19T06:22:25.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-19T06:22:25.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:26.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:26 smithi134 ceph-mon[28283]: Deploying daemon osd.3 on smithi150 2024-01-19T06:22:26.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:26 smithi150 ceph-mon[35328]: Deploying daemon osd.3 on smithi150 2024-01-19T06:22:28.134 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:27 smithi150 ceph-mon[35328]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-01-19T06:22:28.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:27 smithi134 ceph-mon[28283]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-01-19T06:22:30.167 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:29 smithi150 ceph-mon[35328]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-01-19T06:22:30.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:29 smithi134 ceph-mon[28283]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-01-19T06:22:31.581 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:31.581 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:31.581 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:31 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:31.581 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:31 smithi150 ceph-mon[35328]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:31 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:31 smithi134 ceph-mon[28283]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:33.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.530+0000 7f0d8cff9700 1 -- 172.21.15.150:0/3999455837 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0d90102830 con 0x7f0d78068ed0 2024-01-19T06:22:33.530 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 3 on host 'smithi150' 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 msgr2=0x7f0d7806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 0x7f0d7806b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009fd0 tx=0x7f0d8400b040 comp rx=0 tx=0).stop 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 msgr2=0x7f0d900fd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0d80000c00 tx=0x7f0d80009e30 comp rx=0 tx=0).stop 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 shutdown_connections 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0d78068ed0 0x7f0d7806b380 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d900fae50 0x7f0d900fd970 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 --2- 172.21.15.150:0/3999455837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d900fdee0 0x7f0d900fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:33.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 >> 172.21.15.150:0/3999455837 conn(0x7f0d900f68b0 msgr2=0x7f0d90101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:22:33.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 shutdown_connections 2024-01-19T06:22:33.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:33.532+0000 7f0d964ba700 1 -- 172.21.15.150:0/3999455837 wait complete. 2024-01-19T06:22:34.156 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:33 smithi150 ceph-mon[35328]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:34.156 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:33 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:34.156 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:33 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:33 smithi134 ceph-mon[28283]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:33 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:33 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:34.446 DEBUG:teuthology.orchestra.run.smithi150:osd.3> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.3.service 2024-01-19T06:22:34.488 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_3... 2024-01-19T06:22:34.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-19T06:22:34.784 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:22:35.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:34 smithi150 ceph-mon[35328]: from='osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T06:22:35.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:34 smithi150 ceph-mon[35328]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T06:22:35.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:35.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:34 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:35.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:34 smithi134 ceph-mon[28283]: from='osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T06:22:35.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:34 smithi134 ceph-mon[28283]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T06:22:35.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:35.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:34 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: osdmap e24: 4 total, 3 up, 4 in 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: from='osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:22:36.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-mon[35328]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:22:36.088 INFO:journalctl@ceph.osd.3.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-3[40852]: 2024-01-19T06:22:35.984+0000 7fe992656700 -1 osd.3 0 waiting for initial osdmap 2024-01-19T06:22:36.088 INFO:journalctl@ceph.osd.3.smithi150.stdout:Jan 19 06:22:35 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-3[40852]: 2024-01-19T06:22:35.990+0000 7fe98d451700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: osdmap e24: 4 total, 3 up, 4 in 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: from='osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:22:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:35 smithi134 ceph-mon[28283]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:22:37.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:22:37.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: osdmap e25: 4 total, 3 up, 4 in 2024-01-19T06:22:37.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:37.057 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:37.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:37.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] boot 2024-01-19T06:22:37.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: osdmap e26: 4 total, 4 up, 4 in 2024-01-19T06:22:37.058 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012533 s, 837 MB/s 2024-01-19T06:22:37.194 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: osdmap e25: 4 total, 3 up, 4 in 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: osd.3 [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] boot 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: osdmap e26: 4 total, 4 up, 4 in 2024-01-19T06:22:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T06:22:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:37 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:22:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:37 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:22:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:37 smithi150 ceph-mon[35328]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:38.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:37 smithi150 ceph-mon[35328]: osdmap e27: 4 total, 4 up, 4 in 2024-01-19T06:22:38.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-01-19T06:22:38.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:37 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:22:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:37 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:22:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:37 smithi134 ceph-mon[28283]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T06:22:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:37 smithi134 ceph-mon[28283]: osdmap e27: 4 total, 4 up, 4 in 2024-01-19T06:22:39.012 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:22:40.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: osdmap e28: 4 total, 4 up, 4 in 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:40.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:40.184 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:40.184 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:39 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: osdmap e28: 4 total, 4 up, 4 in 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:40.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:40.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:40.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:22:40.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:39 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/1506537270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 msgr2=0x7fbb580fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/1506537270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb580fd5a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbb48004970 tx=0x7fbb48009e30 comp rx=0 tx=0).stop 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/1506537270 shutdown_connections 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/1506537270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 0x7fbb580fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/1506537270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb580fd5a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/1506537270 >> 172.21.15.150:0/1506537270 conn(0x7fbb580f6950 msgr2=0x7fbb580f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/1506537270 shutdown_connections 2024-01-19T06:22:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/1506537270 wait complete. 2024-01-19T06:22:40.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.437+0000 7fbb5e0bc700 1 Processor -- start 2024-01-19T06:22:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb5e0bc700 1 -- start start 2024-01-19T06:22:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb5e0bc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 0x7fbb5810c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb5e0bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb5e0bc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5810cf70 con 0x7fbb580fbe10 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb5e0bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5810d810 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54358/0 (socket says 172.21.15.150:54358) 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 -- 172.21.15.150:0/2212680833 learned_addr learned my addr 172.21.15.150:0/2212680833 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb577fe700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 0x7fbb5810c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 -- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 msgr2=0x7fbb5810c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 0x7fbb5810c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 -- 172.21.15.150:0/2212680833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48005040 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.438+0000 7fbb56ffd700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c002a50 tx=0x7fbb4c0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.439+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb4c007cc0 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.439+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb4c00f040 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.439+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5810daf0 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.439+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb4c00d9c0 con 0x7fbb580fd190 2024-01-19T06:22:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.439+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5810e030 con 0x7fbb580fd190 2024-01-19T06:22:40.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.440+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb58106590 con 0x7fbb580fd190 2024-01-19T06:22:40.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.440+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fbb4c00db20 con 0x7fbb580fd190 2024-01-19T06:22:40.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.440+0000 7fbb3ffff700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 0x7fbb4006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:22:40.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.441+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fbb4c086440 con 0x7fbb580fd190 2024-01-19T06:22:40.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.441+0000 7fbb577fe700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 0x7fbb4006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:22:40.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.441+0000 7fbb577fe700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 0x7fbb4006b410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb48004f50 tx=0x7fbb48009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:22:40.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.445+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbb4c0569c0 con 0x7fbb580fd190 2024-01-19T06:22:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:40.633+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb5810e2c0 con 0x7fbb40068f60 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi150 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi150 to 15936M 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: osdmap e29: 4 total, 4 up, 4 in 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:22:41.254 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:40 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi150 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi150 to 15936M 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: osdmap e29: 4 total, 4 up, 4 in 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:22:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:40 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:42.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:42 smithi150 ceph-mon[35328]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:42.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:42 smithi150 ceph-mon[35328]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:22:42.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:42 smithi150 ceph-mon[35328]: osdmap e30: 4 total, 4 up, 4 in 2024-01-19T06:22:42.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:41 smithi134 ceph-mon[28283]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:42.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:41 smithi134 ceph-mon[28283]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:22:42.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:41 smithi134 ceph-mon[28283]: osdmap e30: 4 total, 4 up, 4 in 2024-01-19T06:22:44.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/1621173109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]: dispatch 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]: dispatch 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]': finished 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: osdmap e31: 5 total, 4 up, 5 in 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:22:44.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:44 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/2468703857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/1621173109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]: dispatch 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]: dispatch 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6efffbf8-04e9-4b01-8723-a6837b2bdb63"}]': finished 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: osdmap e31: 5 total, 4 up, 5 in 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:22:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:44 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2468703857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:22:46.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:46 smithi134 ceph-mon[28283]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-01-19T06:22:46.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:46 smithi150 ceph-mon[35328]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-01-19T06:22:48.160 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:48 smithi150 ceph-mon[35328]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-01-19T06:22:48.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:48 smithi134 ceph-mon[28283]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-01-19T06:22:50.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:50 smithi134 ceph-mon[28283]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-19T06:22:50.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:50 smithi150 ceph-mon[35328]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-19T06:22:51.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:51 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-19T06:22:51.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:51 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:51.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:51 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-19T06:22:51.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:51 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:22:52.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:52 smithi150 ceph-mon[35328]: Deploying daemon osd.4 on smithi150 2024-01-19T06:22:52.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:52 smithi150 ceph-mon[35328]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-19T06:22:52.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:52 smithi134 ceph-mon[28283]: Deploying daemon osd.4 on smithi150 2024-01-19T06:22:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:52 smithi134 ceph-mon[28283]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-19T06:22:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:22:54.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:54 smithi150 ceph-mon[35328]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-19T06:22:54.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:54 smithi134 ceph-mon[28283]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-19T06:22:56.409 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:56 smithi150 ceph-mon[35328]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:56.410 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:56 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:56.410 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:56 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:56.410 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:56 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:56.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:56 smithi134 ceph-mon[28283]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:56.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:56 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:56.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:56 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:22:56.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:56 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:22:58.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:58 smithi150 ceph-mon[35328]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:58.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:58 smithi134 ceph-mon[28283]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:59.298 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2024-01-19T06:22:59.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.297+0000 7fbb3ffff700 1 -- 172.21.15.150:0/2212680833 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbb5810e2c0 con 0x7fbb40068f60 2024-01-19T06:22:59.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:22:59 smithi150 ceph-mon[35328]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 msgr2=0x7fbb4006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 0x7fbb4006b410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbb48004f50 tx=0x7fbb48009e30 comp rx=0 tx=0).stop 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 msgr2=0x7fbb5810c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c002a50 tx=0x7fbb4c0046b0 comp rx=0 tx=0).stop 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 shutdown_connections 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbb40068f60 0x7fbb4006b410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb580fbe10 0x7fbb5810c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 --2- 172.21.15.150:0/2212680833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580fd190 0x7fbb5810c970 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:22:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.299+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 >> 172.21.15.150:0/2212680833 conn(0x7fbb580f6950 msgr2=0x7fbb580f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:22:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.300+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 shutdown_connections 2024-01-19T06:22:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:22:59.300+0000 7fbb5e0bc700 1 -- 172.21.15.150:0/2212680833 wait complete. 2024-01-19T06:22:59.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:22:59 smithi134 ceph-mon[28283]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:22:59.891 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.4.service 2024-01-19T06:22:59.894 INFO:tasks.cephadm:Deploying osd.5 on smithi150 with /dev/vg_nvme/lv_2... 2024-01-19T06:22:59.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.229 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:01.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-19T06:23:01.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: osdmap e32: 5 total, 4 up, 5 in 2024-01-19T06:23:01.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:01.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: from='osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:01.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:01.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-mon[35328]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:23:01.183 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-4[47527]: 2024-01-19T06:23:01.105+0000 7f699dc60700 -1 osd.4 0 waiting for initial osdmap 2024-01-19T06:23:01.183 INFO:journalctl@ceph.osd.4.smithi150.stdout:Jan 19 06:23:01 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-4[47527]: 2024-01-19T06:23:01.112+0000 7f699825a700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: osdmap e32: 5 total, 4 up, 5 in 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: from='osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:01.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:01 smithi134 ceph-mon[28283]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:23:02.280 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:02 smithi150 ceph-mon[35328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:23:02.280 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:02 smithi150 ceph-mon[35328]: osdmap e33: 5 total, 4 up, 5 in 2024-01-19T06:23:02.280 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:02.280 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:02.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:02 smithi134 ceph-mon[28283]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:23:02.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:02 smithi134 ceph-mon[28283]: osdmap e33: 5 total, 4 up, 5 in 2024-01-19T06:23:02.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:02.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] boot 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: osdmap e34: 5 total, 5 up, 5 in 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:03.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:03 smithi134 ceph-mon[28283]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: osd.4 [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] boot 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: osdmap e34: 5 total, 5 up, 5 in 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T06:23:03.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:03 smithi150 ceph-mon[35328]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140331 s, 747 MB/s 2024-01-19T06:23:03.495 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-01-19T06:23:04.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-01-19T06:23:04.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:04 smithi134 ceph-mon[28283]: osdmap e35: 5 total, 5 up, 5 in 2024-01-19T06:23:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:04 smithi150 ceph-mon[35328]: osdmap e35: 5 total, 5 up, 5 in 2024-01-19T06:23:05.014 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi150 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi150 to 7968M 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:05 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 -- 172.21.15.150:0/621332359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080ffdd0 msgr2=0x7fc5081021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:06.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 --2- 172.21.15.150:0/621332359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080ffdd0 0x7fc5081021d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc004940 tx=0x7fc4fc009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:06.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 -- 172.21.15.150:0/621332359 shutdown_connections 2024-01-19T06:23:06.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 --2- 172.21.15.150:0/621332359 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080ffdd0 0x7fc5081021d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 --2- 172.21.15.150:0/621332359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080668c0 0x7fc5080ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 -- 172.21.15.150:0/621332359 >> 172.21.15.150:0/621332359 conn(0x7fc5080f71d0 msgr2=0x7fc5080f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 -- 172.21.15.150:0/621332359 shutdown_connections 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.403+0000 7fc50d27e700 1 -- 172.21.15.150:0/621332359 wait complete. 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 Processor -- start 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 -- start start 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080ffdd0 0x7fc5080fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080fbd90 con 0x7fc5080ffdd0 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080fbf00 con 0x7fc5080668c0 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:06.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42020/0 (socket says 172.21.15.150:42020) 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 -- 172.21.15.150:0/3723566026 learned_addr learned my addr 172.21.15.150:0/3723566026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc5067fc700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080ffdd0 0x7fc5080fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 -- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080ffdd0 msgr2=0x7fc5080fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080ffdd0 0x7fc5080fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 -- 172.21.15.150:0/3723566026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc005040 con 0x7fc5080668c0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc506ffd700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800bbb0 tx=0x7fc4f8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4f80092e0 con 0x7fc5080668c0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4f800cbb0 con 0x7fc5080668c0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.404+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5080fc1e0 con 0x7fc5080668c0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.405+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4f8015d70 con 0x7fc5080668c0 2024-01-19T06:23:06.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.405+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5080fc560 con 0x7fc5080668c0 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.405+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50804bb80 con 0x7fc5080668c0 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.406+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fc4f8012070 con 0x7fc5080668c0 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.406+0000 7fc4ef7fe700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 0x7fc4f006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.406+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fc4f80865b0 con 0x7fc5080668c0 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.407+0000 7fc5067fc700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 0x7fc4f006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:06.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.407+0000 7fc5067fc700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 0x7fc4f006b420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00b730 tx=0x7fc4fc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:06.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.411+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc4f8056980 con 0x7fc5080668c0 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi150 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi150 to 7968M 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:06.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:06.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:06.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:06.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:05 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:06.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:06.599+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc508100a30 con 0x7fc4f0068f70 2024-01-19T06:23:07.281 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:07.281 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:23:07.281 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:23:07.281 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T06:23:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T06:23:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:08.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:07 smithi150 ceph-mon[35328]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:08.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:07 smithi150 ceph-mon[35328]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:23:08.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:07 smithi134 ceph-mon[28283]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:08.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:07 smithi134 ceph-mon[28283]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:23:09.392 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:08 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/4059358252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]: dispatch 2024-01-19T06:23:09.392 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:08 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]: dispatch 2024-01-19T06:23:09.392 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:08 smithi150 ceph-mon[35328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]': finished 2024-01-19T06:23:09.392 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:08 smithi150 ceph-mon[35328]: osdmap e36: 6 total, 5 up, 6 in 2024-01-19T06:23:09.392 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:09.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:08 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/4059358252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]: dispatch 2024-01-19T06:23:09.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:08 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]: dispatch 2024-01-19T06:23:09.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:08 smithi134 ceph-mon[28283]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e42646f-77fc-418a-9339-79b1130d4906"}]': finished 2024-01-19T06:23:09.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:08 smithi134 ceph-mon[28283]: osdmap e36: 6 total, 5 up, 6 in 2024-01-19T06:23:09.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:10.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:10 smithi134 ceph-mon[28283]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:10.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:10 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/2339118101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:23:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:10 smithi150 ceph-mon[35328]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:10 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/2339118101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T06:23:12.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:12 smithi134 ceph-mon[28283]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:12.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:12 smithi150 ceph-mon[35328]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:14.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:14 smithi150 ceph-mon[35328]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:14.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:14 smithi134 ceph-mon[28283]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:16.247 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:16 smithi150 ceph-mon[35328]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:16.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:16 smithi134 ceph-mon[28283]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:17.345 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-19T06:23:17.345 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:17 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-19T06:23:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:17 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:18.277 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:18 smithi150 ceph-mon[35328]: Deploying daemon osd.5 on smithi150 2024-01-19T06:23:18.277 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:18 smithi150 ceph-mon[35328]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:18 smithi134 ceph-mon[28283]: Deploying daemon osd.5 on smithi150 2024-01-19T06:23:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:18 smithi134 ceph-mon[28283]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:20.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:20 smithi150 ceph-mon[35328]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:20.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:20 smithi134 ceph-mon[28283]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:22.396 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:22 smithi150 ceph-mon[35328]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:22.396 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:22.396 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:22.396 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:22.396 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:22 smithi134 ceph-mon[28283]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:24.346 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:24 smithi150 ceph-mon[35328]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:24.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.411+0000 7fc4ef7fe700 1 -- 172.21.15.150:0/3723566026 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc508100a30 con 0x7fc4f0068f70 2024-01-19T06:23:24.411 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 5 on host 'smithi150' 2024-01-19T06:23:24.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.413+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 msgr2=0x7fc4f006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 0x7fc4f006b420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00b730 tx=0x7fc4fc009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 msgr2=0x7fc5080fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800bbb0 tx=0x7fc4f8009840 comp rx=0 tx=0).stop 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 shutdown_connections 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc4f0068f70 0x7fc4f006b420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:24.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5080668c0 0x7fc5080fd280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:24.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 --2- 172.21.15.150:0/3723566026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5080ffdd0 0x7fc5080fb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:24.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 >> 172.21.15.150:0/3723566026 conn(0x7fc5080f71d0 msgr2=0x7fc5080f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:24.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 shutdown_connections 2024-01-19T06:23:24.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:24.414+0000 7fc50d27e700 1 -- 172.21.15.150:0/3723566026 wait complete. 2024-01-19T06:23:24.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:24 smithi134 ceph-mon[28283]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:25.279 DEBUG:teuthology.orchestra.run.smithi150:osd.5> sudo journalctl -f -n 0 -u ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.5.service 2024-01-19T06:23:25.281 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-01-19T06:23:25.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd stat -f json 2024-01-19T06:23:25.551 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:25 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:25 smithi134 ceph-mon[28283]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:25 smithi134 ceph-mon[28283]: from='osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T06:23:25.673 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:25 smithi134 ceph-mon[28283]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T06:23:25.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:25.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:25 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:25.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:25 smithi150 ceph-mon[35328]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:25.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:25 smithi150 ceph-mon[35328]: from='osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T06:23:25.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:25 smithi150 ceph-mon[35328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: osdmap e37: 6 total, 5 up, 6 in 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:26.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:26 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:26.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-19T06:23:26.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: osdmap e37: 6 total, 5 up, 6 in 2024-01-19T06:23:26.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:26.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:26.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-01-19T06:23:26.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:26.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:26.683 INFO:journalctl@ceph.osd.5.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-5[54441]: 2024-01-19T06:23:26.394+0000 7fa5dde08700 -1 osd.5 0 waiting for initial osdmap 2024-01-19T06:23:26.683 INFO:journalctl@ceph.osd.5.smithi150.stdout:Jan 19 06:23:26 smithi150 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-5[54441]: 2024-01-19T06:23:26.403+0000 7fa5d7400700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 -- 172.21.15.134:0/726029475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fde00 msgr2=0x7fe1ec0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/726029475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec0fe250 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4004940 tx=0x7fe1d4009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 -- 172.21.15.134:0/726029475 shutdown_connections 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/726029475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec0fe250 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/726029475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 -- 172.21.15.134:0/726029475 >> 172.21.15.134:0/726029475 conn(0x7fe1ec0f82d0 msgr2=0x7fe1ec0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 -- 172.21.15.134:0/726029475 shutdown_connections 2024-01-19T06:23:27.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.389+0000 7fe1f1cef700 1 -- 172.21.15.134:0/726029475 wait complete. 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 Processor -- start 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 -- start start 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec10fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec110330 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1f1cef700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1ec110b70 con 0x7fe1ec0fde00 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1eb7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37560/0 (socket says 172.21.15.134:37560) 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1eb7fe700 1 -- 172.21.15.134:0/2391544859 learned_addr learned my addr 172.21.15.134:0/2391544859 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.390+0000 7fe1eaffd700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec10fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1eb7fe700 1 -- 172.21.15.134:0/2391544859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fde00 msgr2=0x7fe1ec10fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1eb7fe700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec10fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1eb7fe700 1 -- 172.21.15.134:0/2391544859 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4005040 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1eb7fe700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400bae0 tx=0x7fe1e4009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e400c840 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1ec110e50 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1e400ce50 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1ec111390 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.391+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e40073d0 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.392+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1ec101ea0 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.393+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fe1e4014410 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.394+0000 7fe1e3fff700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 0x7fe1d806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.394+0000 7fe1eaffd700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 0x7fe1d806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.394+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fe1e4086a30 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.394+0000 7fe1eaffd700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 0x7fe1d806b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4004890 tx=0x7fe1d4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:27.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.400+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe1e404ca00 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.598+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe1ec04bb80 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.598+0000 7fe1e3fff700 1 -- 172.21.15.134:0/2391544859 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fe1e4056cc0 con 0x7fe1ec0fcbf0 2024-01-19T06:23:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 msgr2=0x7fe1d806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 0x7fe1d806b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4004890 tx=0x7fe1d4009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 msgr2=0x7fe1ec10f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400bae0 tx=0x7fe1e4009810 comp rx=0 tx=0).stop 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 shutdown_connections 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe1d8068f60 0x7fe1d806b410 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1ec0fcbf0 0x7fe1ec10f760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 --2- 172.21.15.134:0/2391544859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec0fde00 0x7fe1ec10fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:27.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 >> 172.21.15.134:0/2391544859 conn(0x7fe1ec0f82d0 msgr2=0x7fe1ec0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:27.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 shutdown_connections 2024-01-19T06:23:27.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:27.600+0000 7fe1f1cef700 1 -- 172.21.15.134:0/2391544859 wait complete. 2024-01-19T06:23:27.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:27 smithi134 ceph-mon[28283]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:23:27.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:27 smithi134 ceph-mon[28283]: osdmap e38: 6 total, 5 up, 6 in 2024-01-19T06:23:27.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:27.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:27.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:27 smithi134 ceph-mon[28283]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:27.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:27 smithi150 ceph-mon[35328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-01-19T06:23:27.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:27 smithi150 ceph-mon[35328]: osdmap e38: 6 total, 5 up, 6 in 2024-01-19T06:23:27.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:27.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:27.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:27 smithi150 ceph-mon[35328]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T06:23:28.352 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1705645407,"num_in_osds":6,"osd_in_since":1705645388,"num_remapped_pgs":0} 2024-01-19T06:23:28.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T06:23:28.614 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: purged_snaps scrub starts 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: purged_snaps scrub ok 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] boot 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: osdmap e39: 6 total, 6 up, 6 in 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:28.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:28 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2391544859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: purged_snaps scrub starts 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: purged_snaps scrub ok 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: osd.5 [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] boot 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: osdmap e39: 6 total, 6 up, 6 in 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T06:23:28.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:28 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2391544859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: osdmap e40: 6 total, 6 up, 6 in 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:29 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: osdmap e40: 6 total, 6 up, 6 in 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-19T06:23:29.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:29.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:29.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:29.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:29 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:23:30.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 -- 172.21.15.134:0/3677390788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 msgr2=0x7fdeb00fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:30.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/3677390788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb00fbfb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdea4004940 tx=0x7fdea4009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:30.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 -- 172.21.15.134:0/3677390788 shutdown_connections 2024-01-19T06:23:30.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/3677390788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb00fbfb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/3677390788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 -- 172.21.15.134:0/3677390788 >> 172.21.15.134:0/3677390788 conn(0x7fdeb00f76c0 msgr2=0x7fdeb00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 -- 172.21.15.134:0/3677390788 shutdown_connections 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.271+0000 7fdeb5c15700 1 -- 172.21.15.134:0/3677390788 wait complete. 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 Processor -- start 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 -- start start 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0114240 con 0x7fdeb00fbb40 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeb5c15700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb010d530 con 0x7fdeb0101b00 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeaf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeaf7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37576/0 (socket says 172.21.15.134:37576) 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.272+0000 7fdeaf7fe700 1 -- 172.21.15.134:0/2440665135 learned_addr learned my addr 172.21.15.134:0/2440665135 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaeffd700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaf7fe700 1 -- 172.21.15.134:0/2440665135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 msgr2=0x7fdeb0113bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaf7fe700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0113bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaf7fe700 1 -- 172.21.15.134:0/2440665135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4005040 con 0x7fdeb00fbb40 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaeffd700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeaf7fe700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdea000b790 tx=0x7fdea0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea000f770 con 0x7fdeb00fbb40 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb010d810 con 0x7fdeb00fbb40 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdea000fd80 con 0x7fdeb00fbb40 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea000d310 con 0x7fdeb00fbb40 2024-01-19T06:23:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.273+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb010dd50 con 0x7fdeb00fbb40 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.274+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb0113360 con 0x7fdeb00fbb40 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.274+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fdea000f8d0 con 0x7fdeb00fbb40 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.275+0000 7fde97fff700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.275+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdea0086a60 con 0x7fdeb00fbb40 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.275+0000 7fdeaeffd700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.275+0000 7fdeaeffd700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 0x7fde9806b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdea4004560 tx=0x7fdea4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:30.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.279+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdea0056d00 con 0x7fdeb00fbb40 2024-01-19T06:23:30.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.465+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb0103fb0 con 0x7fdeb00fbb40 2024-01-19T06:23:30.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.466+0000 7fde97fff700 1 -- 172.21.15.134:0/2440665135 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7fdea0056890 con 0x7fdeb00fbb40 2024-01-19T06:23:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:23:30.466 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":40,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","created":"2024-01-19T06:18:02.238207+0000","modified":"2024-01-19T06:23:28.402456+0000","last_up_change":"2024-01-19T06:23:27.392310+0000","last_in_change":"2024-01-19T06:23:08.932614+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T06:22:14.846038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6866d2e8-d629-4fd8-ab59-37109281459b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6803","nonce":3497975570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6805","nonce":3497975570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497975570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6807","nonce":3497975570}]},"public_addr":"172.21.15.134:6803/3497975570","cluster_addr":"172.21.15.134:6805/3497975570","heartbeat_back_addr":"172.21.15.134:6809/3497975570","heartbeat_front_addr":"172.21.15.134:6807/3497975570","state":["exists","up"]},{"osd":1,"uuid":"c8d35ff7-48c7-49f5-afa6-e43f5d10212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6811","nonce":743565436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6813","nonce":743565436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6817","nonce":743565436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6815","nonce":743565436}]},"public_addr":"172.21.15.134:6811/743565436","cluster_addr":"172.21.15.134:6813/743565436","heartbeat_back_addr":"172.21.15.134:6817/743565436","heartbeat_front_addr":"172.21.15.134:6815/743565436","state":["exists","up"]},{"osd":2,"uuid":"b2df8d4e-2a3c-479a-8033-2b777cf50888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6819","nonce":3103353719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6821","nonce":3103353719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6825","nonce":3103353719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6823","nonce":3103353719}]},"public_addr":"172.21.15.134:6819/3103353719","cluster_addr":"172.21.15.134:6821/3103353719","heartbeat_back_addr":"172.21.15.134:6825/3103353719","heartbeat_front_addr":"172.21.15.134:6823/3103353719","state":["exists","up"]},{"osd":3,"uuid":"b66525f9-4493-4708-bbf7-e76de91fd5b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6801","nonce":4178102778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6803","nonce":4178102778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6807","nonce":4178102778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6805","nonce":4178102778}]},"public_addr":"172.21.15.150:6801/4178102778","cluster_addr":"172.21.15.150:6803/4178102778","heartbeat_back_addr":"172.21.15.150:6807/4178102778","heartbeat_front_addr":"172.21.15.150:6805/4178102778","state":["exists","up"]},{"osd":4,"uuid":"6efffbf8-04e9-4b01-8723-a6837b2bdb63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6809","nonce":996513041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6811","nonce":996513041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6815","nonce":996513041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6813","nonce":996513041}]},"public_addr":"172.21.15.150:6809/996513041","cluster_addr":"172.21.15.150:6811/996513041","heartbeat_back_addr":"172.21.15.150:6815/996513041","heartbeat_front_addr":"172.21.15.150:6813/996513041","state":["exists","up"]},{"osd":5,"uuid":"8e42646f-77fc-418a-9339-79b1130d4906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6817","nonce":2546848737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6819","nonce":2546848737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6823","nonce":2546848737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6821","nonce":2546848737}]},"public_addr":"172.21.15.150:6817/2546848737","cluster_addr":"172.21.15.150:6819/2546848737","heartbeat_back_addr":"172.21.15.150:6823/2546848737","heartbeat_front_addr":"172.21.15.150:6821/2546848737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:16.885118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:45.420119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:12.603236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:35.079173+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:00.172610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:26.088382+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6801/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3213922814":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/2127635670":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1113248930":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/2486479390":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1811685245":"2024-01-20T06:18:33.202902+0000","172.21.15.134:6800/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1277951871":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1593458977":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3378221656":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/620851043":"2024-01-20T06:18:33.202902+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 msgr2=0x7fde9806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 0x7fde9806b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdea4004560 tx=0x7fdea4009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 msgr2=0x7fdeb01136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdea000b790 tx=0x7fdea0009250 comp rx=0 tx=0).stop 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 shutdown_connections 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdeb00fbb40 0x7fdeb01136b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 --2- 172.21.15.134:0/2440665135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb0101b00 0x7fdeb0113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:30.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 >> 172.21.15.134:0/2440665135 conn(0x7fdeb00f76c0 msgr2=0x7fdeb00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:30.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 shutdown_connections 2024-01-19T06:23:30.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:30.468+0000 7fdeb5c15700 1 -- 172.21.15.134:0/2440665135 wait complete. 2024-01-19T06:23:30.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:30 smithi134 ceph-mon[28283]: Detected new or changed devices on smithi150 2024-01-19T06:23:30.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:30 smithi134 ceph-mon[28283]: Adjusting osd_memory_target on smithi150 to 5312M 2024-01-19T06:23:30.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:30 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:30.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:30 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:30.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:30 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:30.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:30 smithi150 ceph-mon[35328]: Detected new or changed devices on smithi150 2024-01-19T06:23:30.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:30 smithi150 ceph-mon[35328]: Adjusting osd_memory_target on smithi150 to 5312M 2024-01-19T06:23:30.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:30 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:23:30.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:30 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:23:30.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:30 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:23:31.062 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-19T06:22:14.846038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-19T06:23:31.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-01-19T06:23:31.316 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:31 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2440665135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:31.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:31 smithi134 ceph-mon[28283]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:31.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:31 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2440665135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:31.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:31 smithi150 ceph-mon[35328]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:32.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- 172.21.15.134:0/300152898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e140fbb40 msgr2=0x7f6e140fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:32.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/300152898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e140fbb40 0x7f6e140fbfb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6e04004940 tx=0x7f6e04009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- 172.21.15.134:0/300152898 shutdown_connections 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/300152898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e140fbb40 0x7f6e140fbfb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/300152898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e14101b00 0x7f6e14101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- 172.21.15.134:0/300152898 >> 172.21.15.134:0/300152898 conn(0x7f6e140f76c0 msgr2=0x7f6e140f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- 172.21.15.134:0/300152898 shutdown_connections 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- 172.21.15.134:0/300152898 wait complete. 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 Processor -- start 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.987+0000 7f6e1b04c700 1 -- start start 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e140fbb40 0x7f6e1410e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1410f210 con 0x7f6e14101b00 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1410f380 con 0x7f6e140fbb40 2024-01-19T06:23:32.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37596/0 (socket says 172.21.15.134:37596) 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 -- 172.21.15.134:0/3580780841 learned_addr learned my addr 172.21.15.134:0/3580780841 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e18de8700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e140fbb40 0x7f6e1410e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 -- 172.21.15.134:0/3580780841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e140fbb40 msgr2=0x7f6e1410e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e140fbb40 0x7f6e1410e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 -- 172.21.15.134:0/3580780841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04005040 con 0x7f6e14101b00 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e0bfff700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009360 tx=0x7f6e04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0400e040 con 0x7f6e14101b00 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e04013420 con 0x7f6e14101b00 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14117670 con 0x7f6e14101b00 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0401c610 con 0x7f6e14101b00 2024-01-19T06:23:32.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.988+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14117bb0 con 0x7f6e14101b00 2024-01-19T06:23:32.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.990+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e14103fb0 con 0x7f6e14101b00 2024-01-19T06:23:32.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.990+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f6e0400bb80 con 0x7f6e14101b00 2024-01-19T06:23:32.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.990+0000 7f6e097fa700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 0x7f6dfc06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:32.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.990+0000 7f6e18de8700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 0x7f6dfc06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:32.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.990+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6e04087c00 con 0x7f6e14101b00 2024-01-19T06:23:32.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.991+0000 7f6e18de8700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 0x7f6dfc06b420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e10009bf0 tx=0x7f6e100091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:32.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:32.995+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6e0408d020 con 0x7f6e14101b00 2024-01-19T06:23:33.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.178+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e14117800 con 0x7f6e14101b00 2024-01-19T06:23:33.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.178+0000 7f6e097fa700 1 -- 172.21.15.134:0/3580780841 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e14117800 con 0x7f6e14101b00 2024-01-19T06:23:33.178 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 msgr2=0x7f6dfc06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 0x7f6dfc06b420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e10009bf0 tx=0x7f6e100091a0 comp rx=0 tx=0).stop 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 msgr2=0x7f6e1410ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009360 tx=0x7f6e04009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 shutdown_connections 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dfc068f70 0x7f6dfc06b420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:33.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e140fbb40 0x7f6e1410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:33.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 --2- 172.21.15.134:0/3580780841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e14101b00 0x7f6e1410ebf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:33.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 >> 172.21.15.134:0/3580780841 conn(0x7f6e140f76c0 msgr2=0x7f6e140f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:33.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 shutdown_connections 2024-01-19T06:23:33.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:33.180+0000 7f6e1b04c700 1 -- 172.21.15.134:0/3580780841 wait complete. 2024-01-19T06:23:33.826 INFO:tasks.cephadm:Setting up client nodes... 2024-01-19T06:23:33.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-19T06:23:34.083 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:34.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:33 smithi134 ceph-mon[28283]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:34.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:33 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3580780841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T06:23:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:33 smithi150 ceph-mon[35328]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:33 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3580780841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 -- 172.21.15.134:0/174249645 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200fb950 msgr2=0x7f46200fe200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 --2- 172.21.15.134:0/174249645 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200fb950 0x7f46200fe200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4610004940 tx=0x7f4610009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 -- 172.21.15.134:0/174249645 shutdown_connections 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 --2- 172.21.15.134:0/174249645 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200fb950 0x7f46200fe200 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 --2- 172.21.15.134:0/174249645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46200faf70 0x7f46200fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 -- 172.21.15.134:0/174249645 >> 172.21.15.134:0/174249645 conn(0x7f46200f6950 msgr2=0x7f46200f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 -- 172.21.15.134:0/174249645 shutdown_connections 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.791+0000 7f46283c3700 1 -- 172.21.15.134:0/174249645 wait complete. 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 Processor -- start 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 -- start start 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46200fbf50 0x7f46200fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:35.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46201114f0 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f46283c3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620111630 con 0x7f46200fbf50 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37616/0 (socket says 172.21.15.134:37616) 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 -- 172.21.15.134:0/3524846410 learned_addr learned my addr 172.21.15.134:0/3524846410 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 -- 172.21.15.134:0/3524846410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46200fbf50 msgr2=0x7f46200fc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46200fbf50 0x7f46200fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 -- 172.21.15.134:0/3524846410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610005040 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f462615f700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b7c0 tx=0x7f461c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.792+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461c00f6f0 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.793+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46200fca20 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.793+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f461c00fd00 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.793+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461c011e90 con 0x7f46200faf70 2024-01-19T06:23:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.793+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462011a010 con 0x7f46200faf70 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.794+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462004bb80 con 0x7f46200faf70 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.794+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f461c00f850 con 0x7f46200faf70 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.794+0000 7f4616ffd700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 0x7f460c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.794+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f461c086910 con 0x7f46200faf70 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.794+0000 7f462595e700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 0x7f460c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.795+0000 7f462595e700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 0x7f460c06b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4610004f90 tx=0x7f4610009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:35.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:35.799+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f461c056bb0 con 0x7f46200faf70 2024-01-19T06:23:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.057+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f46200659e0 con 0x7f46200faf70 2024-01-19T06:23:36.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.063+0000 7f4616ffd700 1 -- 172.21.15.134:0/3524846410 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f461c056740 con 0x7f46200faf70 2024-01-19T06:23:36.063 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-01-19T06:23:36.063 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBoFaplmASIAxAA0STPTx8TL/0ek+O+pH0KPA== 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 msgr2=0x7f460c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 0x7f460c06b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4610004f90 tx=0x7f4610009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 msgr2=0x7f46200fdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f461c00b7c0 tx=0x7f461c009250 comp rx=0 tx=0).stop 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 shutdown_connections 2024-01-19T06:23:36.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f460c068f60 0x7f460c06b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:36.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46200faf70 0x7f46200fdad0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:36.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 --2- 172.21.15.134:0/3524846410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46200fbf50 0x7f46200fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:36.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 >> 172.21.15.134:0/3524846410 conn(0x7f46200f6950 msgr2=0x7f46201013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:36.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 shutdown_connections 2024-01-19T06:23:36.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:36.065+0000 7f46283c3700 1 -- 172.21.15.134:0/3524846410 wait complete. 2024-01-19T06:23:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:35 smithi134 ceph-mon[28283]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:35 smithi150 ceph-mon[35328]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:36.753 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:23:36.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-01-19T06:23:36.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-19T06:23:36.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-19T06:23:37.063 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi150/config 2024-01-19T06:23:37.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:36 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3524846410' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:37.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:36 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3524846410' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T06:23:37.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:36 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:36 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3524846410' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:36 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3524846410' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T06:23:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:36 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:38.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:37 smithi134 ceph-mon[28283]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:38.267 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:37 smithi150 ceph-mon[35328]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 -- 172.21.15.150:0/778619132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fde00 msgr2=0x7fe54c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 --2- 172.21.15.150:0/778619132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fde00 0x7fe54c0fe250 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe540004940 tx=0x7fe540009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 -- 172.21.15.150:0/778619132 shutdown_connections 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 --2- 172.21.15.150:0/778619132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fde00 0x7fe54c0fe250 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 --2- 172.21.15.150:0/778619132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 -- 172.21.15.150:0/778619132 >> 172.21.15.150:0/778619132 conn(0x7fe54c0f82d0 msgr2=0x7fe54c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 -- 172.21.15.150:0/778619132 shutdown_connections 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.541+0000 7fe551a46700 1 -- 172.21.15.150:0/778619132 wait complete. 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 Processor -- start 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 -- start start 2024-01-19T06:23:38.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fde00 0x7fe54c10da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10e050 con 0x7fe54c0fde00 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe551a46700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10e8f0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe54affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe54affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42192/0 (socket says 172.21.15.150:42192) 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe54affd700 1 -- 172.21.15.150:0/3416359719 learned_addr learned my addr 172.21.15.150:0/3416359719 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.542+0000 7fe54a7fc700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fde00 0x7fe54c10da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe54affd700 1 -- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fde00 msgr2=0x7fe54c10da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe54affd700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fde00 0x7fe54c10da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe54affd700 1 -- 172.21.15.150:0/3416359719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe540005040 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe54affd700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe53c002a50 tx=0x7fe53c004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c007cc0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe53c00f040 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c10ebd0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe53c00d8d0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.543+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c10f110 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.544+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c04bb80 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.544+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fe53c00db40 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.545+0000 7fe5337fe700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 0x7fe53406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.545+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe53c0867e0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.545+0000 7fe54a7fc700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 0x7fe53406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:38.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.545+0000 7fe54a7fc700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 0x7fe53406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe540004890 tx=0x7fe540009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:38.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.549+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe53c056a00 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.807+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe54c1055b0 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.812+0000 7fe5337fe700 1 -- 172.21.15.150:0/3416359719 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe53c056590 con 0x7fe54c0fcbf0 2024-01-19T06:23:38.812 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-01-19T06:23:38.812 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBqFaplzR8/MBAAI/+MYpukWkHSXWtO64OflQ== 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 msgr2=0x7fe53406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 0x7fe53406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe540004890 tx=0x7fe540009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 msgr2=0x7fe54c10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe53c002a50 tx=0x7fe53c004600 comp rx=0 tx=0).stop 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 shutdown_connections 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fe534068f60 0x7fe53406b410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe54c0fcbf0 0x7fe54c10d510 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 --2- 172.21.15.150:0/3416359719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe54c0fde00 0x7fe54c10da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 >> 172.21.15.150:0/3416359719 conn(0x7fe54c0f82d0 msgr2=0x7fe54c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 shutdown_connections 2024-01-19T06:23:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-01-19T06:23:38.814+0000 7fe551a46700 1 -- 172.21.15.150:0/3416359719 wait complete. 2024-01-19T06:23:39.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:38 smithi150 ceph-mon[35328]: from='client.? 172.21.15.150:0/3416359719' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:39.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:38 smithi150 ceph-mon[35328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:39.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:38 smithi150 ceph-mon[35328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T06:23:39.412 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:23:39.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-01-19T06:23:39.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-19T06:23:39.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:38 smithi134 ceph-mon[28283]: from='client.? 172.21.15.150:0/3416359719' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:39.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:38 smithi134 ceph-mon[28283]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T06:23:39.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:38 smithi134 ceph-mon[28283]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T06:23:39.454 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-19T06:23:39.454 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-01-19T06:23:39.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph mgr dump --format=json 2024-01-19T06:23:39.720 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:40.115 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:39 smithi134 ceph-mon[28283]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:40.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:39 smithi150 ceph-mon[35328]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:41.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 -- 172.21.15.134:0/2291076369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 msgr2=0x7fce5c0fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 --2- 172.21.15.134:0/2291076369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fdfe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce4c004940 tx=0x7fce4c009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 -- 172.21.15.134:0/2291076369 shutdown_connections 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 --2- 172.21.15.134:0/2291076369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 --2- 172.21.15.134:0/2291076369 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fdfe0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 -- 172.21.15.134:0/2291076369 >> 172.21.15.134:0/2291076369 conn(0x7fce5c0f76c0 msgr2=0x7fce5c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 -- 172.21.15.134:0/2291076369 shutdown_connections 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 -- 172.21.15.134:0/2291076369 wait complete. 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.462+0000 7fce60ca1700 1 Processor -- start 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 -- start start 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c0fbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c0fc580 con 0x7fce5c0fdc10 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c0fc6f0 con 0x7fce5c0fe5b0 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39992/0 (socket says 172.21.15.134:39992) 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 -- 172.21.15.134:0/988832890 learned_addr learned my addr 172.21.15.134:0/988832890 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce59d9b700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c0fbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 -- 172.21.15.134:0/988832890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 msgr2=0x7fce5c0fbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c0fbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 -- 172.21.15.134:0/988832890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c005040 con 0x7fce5c0fdc10 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce59d9b700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c0fbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce5a59c700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00b3b0 tx=0x7fce4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4c00e040 con 0x7fce5c0fdc10 2024-01-19T06:23:41.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c119870 con 0x7fce5c0fdc10 2024-01-19T06:23:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce4c004290 con 0x7fce5c0fdc10 2024-01-19T06:23:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4c01c3f0 con 0x7fce5c0fdc10 2024-01-19T06:23:41.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.463+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c119cd0 con 0x7fce5c0fdc10 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.464+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c104200 con 0x7fce5c0fdc10 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.465+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fce4c00bc50 con 0x7fce5c0fdc10 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.465+0000 7fce4affd700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 0x7fce4406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.465+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fce4c087f00 con 0x7fce5c0fdc10 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.465+0000 7fce59d9b700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 0x7fce4406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:41.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.466+0000 7fce59d9b700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 0x7fce4406b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fce50009fd0 tx=0x7fce5000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:41.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.470+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce4c054710 con 0x7fce5c0fdc10 2024-01-19T06:23:41.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.705+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fce5c0fb2c0 con 0x7fce5c0fdc10 2024-01-19T06:23:41.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.708+0000 7fce4affd700 1 -- 172.21.15.134:0/988832890 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166780 (secure 0 0 0) 0x7fce4c057d30 con 0x7fce5c0fdc10 2024-01-19T06:23:41.710 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 msgr2=0x7fce4406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 0x7fce4406b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fce50009fd0 tx=0x7fce5000b040 comp rx=0 tx=0).stop 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 msgr2=0x7fce5c0fba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00b3b0 tx=0x7fce4c009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 shutdown_connections 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fce44068f60 0x7fce4406b410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce5c0fdc10 0x7fce5c0fba40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 --2- 172.21.15.134:0/988832890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0fe5b0 0x7fce5c0fbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 >> 172.21.15.134:0/988832890 conn(0x7fce5c0f76c0 msgr2=0x7fce5c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 shutdown_connections 2024-01-19T06:23:41.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:41.711+0000 7fce60ca1700 1 -- 172.21.15.134:0/988832890 wait complete. 2024-01-19T06:23:42.410 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:41 smithi134 ceph-mon[28283]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:42.410 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:41 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/988832890' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-19T06:23:42.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:41 smithi150 ceph-mon[35328]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:42.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:41 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/988832890' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-19T06:23:42.435 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi134.qrxokr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":233880040},{"type":"v1","addr":"172.21.15.134:6801","nonce":233880040}]},"active_addr":"172.21.15.134:6801/233880040","active_change":"2024-01-19T06:20:06.516991+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14226,"name":"smithi150.xkxrco","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.134:8443/","prometheus":"http://172.21.15.134:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":131047033}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":745400049}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":531128544}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1116745205}]}]} 2024-01-19T06:23:42.438 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-01-19T06:23:42.438 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-01-19T06:23:42.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T06:23:42.695 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:44.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:43 smithi134 ceph-mon[28283]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:44.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:43 smithi150 ceph-mon[35328]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 -- 172.21.15.134:0/1341998394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 msgr2=0x7f9498102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 --2- 172.21.15.134:0/1341998394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498102fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f948c004940 tx=0x7f948c009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 -- 172.21.15.134:0/1341998394 shutdown_connections 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 --2- 172.21.15.134:0/1341998394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 0x7f94980fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 --2- 172.21.15.134:0/1341998394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498102fd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.409+0000 7f949f037700 1 -- 172.21.15.134:0/1341998394 >> 172.21.15.134:0/1341998394 conn(0x7f94980f8790 msgr2=0x7f94980faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.410+0000 7f949f037700 1 -- 172.21.15.134:0/1341998394 shutdown_connections 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.410+0000 7f949f037700 1 -- 172.21.15.134:0/1341998394 wait complete. 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.410+0000 7f949f037700 1 Processor -- start 2024-01-19T06:23:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.410+0000 7f949f037700 1 -- start start 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 0x7f9498115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9498116470 con 0x7f9498102c00 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949810f710 con 0x7f94981035a0 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:44.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40018/0 (socket says 172.21.15.134:40018) 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 -- 172.21.15.134:0/1324506032 learned_addr learned my addr 172.21.15.134:0/1324506032 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f9497fff700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 0x7f9498115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 -- 172.21.15.134:0/1324506032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 msgr2=0x7f9498115d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 0x7f9498115d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 -- 172.21.15.134:0/1324506032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c005040 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949cdd3700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f948c009180 tx=0x7f948c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f948c00e040 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f948c004290 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949810f990 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f948c01c3f0 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.411+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949810fed0 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.412+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980fc390 con 0x7f9498102c00 2024-01-19T06:23:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.413+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f948c01c6b0 con 0x7f9498102c00 2024-01-19T06:23:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.413+0000 7f94957fa700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 0x7f948806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.413+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f948c087d30 con 0x7f9498102c00 2024-01-19T06:23:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.413+0000 7f9497fff700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 0x7f948806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.414+0000 7f9497fff700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 0x7f948806b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9498111080 tx=0x7f94840091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:44.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.417+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f948c058120 con 0x7f9498102c00 2024-01-19T06:23:44.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.603+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9498110e30 con 0x7f9498102c00 2024-01-19T06:23:44.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.604+0000 7f94957fa700 1 -- 172.21.15.134:0/1324506032 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7f948c057cb0 con 0x7f9498102c00 2024-01-19T06:23:44.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:23:44.604 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":40,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","created":"2024-01-19T06:18:02.238207+0000","modified":"2024-01-19T06:23:28.402456+0000","last_up_change":"2024-01-19T06:23:27.392310+0000","last_in_change":"2024-01-19T06:23:08.932614+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T06:22:14.846038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6866d2e8-d629-4fd8-ab59-37109281459b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6803","nonce":3497975570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6805","nonce":3497975570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497975570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6807","nonce":3497975570}]},"public_addr":"172.21.15.134:6803/3497975570","cluster_addr":"172.21.15.134:6805/3497975570","heartbeat_back_addr":"172.21.15.134:6809/3497975570","heartbeat_front_addr":"172.21.15.134:6807/3497975570","state":["exists","up"]},{"osd":1,"uuid":"c8d35ff7-48c7-49f5-afa6-e43f5d10212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6811","nonce":743565436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6813","nonce":743565436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6817","nonce":743565436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6815","nonce":743565436}]},"public_addr":"172.21.15.134:6811/743565436","cluster_addr":"172.21.15.134:6813/743565436","heartbeat_back_addr":"172.21.15.134:6817/743565436","heartbeat_front_addr":"172.21.15.134:6815/743565436","state":["exists","up"]},{"osd":2,"uuid":"b2df8d4e-2a3c-479a-8033-2b777cf50888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6819","nonce":3103353719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6821","nonce":3103353719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6825","nonce":3103353719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6823","nonce":3103353719}]},"public_addr":"172.21.15.134:6819/3103353719","cluster_addr":"172.21.15.134:6821/3103353719","heartbeat_back_addr":"172.21.15.134:6825/3103353719","heartbeat_front_addr":"172.21.15.134:6823/3103353719","state":["exists","up"]},{"osd":3,"uuid":"b66525f9-4493-4708-bbf7-e76de91fd5b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6801","nonce":4178102778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6803","nonce":4178102778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6807","nonce":4178102778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6805","nonce":4178102778}]},"public_addr":"172.21.15.150:6801/4178102778","cluster_addr":"172.21.15.150:6803/4178102778","heartbeat_back_addr":"172.21.15.150:6807/4178102778","heartbeat_front_addr":"172.21.15.150:6805/4178102778","state":["exists","up"]},{"osd":4,"uuid":"6efffbf8-04e9-4b01-8723-a6837b2bdb63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6809","nonce":996513041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6811","nonce":996513041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6815","nonce":996513041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6813","nonce":996513041}]},"public_addr":"172.21.15.150:6809/996513041","cluster_addr":"172.21.15.150:6811/996513041","heartbeat_back_addr":"172.21.15.150:6815/996513041","heartbeat_front_addr":"172.21.15.150:6813/996513041","state":["exists","up"]},{"osd":5,"uuid":"8e42646f-77fc-418a-9339-79b1130d4906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6817","nonce":2546848737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6819","nonce":2546848737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6823","nonce":2546848737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6821","nonce":2546848737}]},"public_addr":"172.21.15.150:6817/2546848737","cluster_addr":"172.21.15.150:6819/2546848737","heartbeat_back_addr":"172.21.15.150:6823/2546848737","heartbeat_front_addr":"172.21.15.150:6821/2546848737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:16.885118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:45.420119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:12.603236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:35.079173+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:00.172610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:26.088382+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6801/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3213922814":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/2127635670":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1113248930":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/2486479390":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1811685245":"2024-01-20T06:18:33.202902+0000","172.21.15.134:6800/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1277951871":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1593458977":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3378221656":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/620851043":"2024-01-20T06:18:33.202902+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T06:23:44.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 msgr2=0x7f948806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:44.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 0x7f948806b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9498111080 tx=0x7f94840091a0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 msgr2=0x7f9498115850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:44.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f948c009180 tx=0x7f948c009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:44.606 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 shutdown_connections 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9488068f60 0x7f948806b410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9498102c00 0x7f9498115850 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 --2- 172.21.15.134:0/1324506032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94981035a0 0x7f9498115d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 >> 172.21.15.134:0/1324506032 conn(0x7f94980f8790 msgr2=0x7f94980fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 shutdown_connections 2024-01-19T06:23:44.607 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:44.606+0000 7f949f037700 1 -- 172.21.15.134:0/1324506032 wait complete. 2024-01-19T06:23:45.358 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:45 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1324506032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:45.389 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-01-19T06:23:45.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T06:23:45.428 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:45 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1324506032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:45.650 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:46.027 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:46 smithi134 ceph-mon[28283]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:46.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:46 smithi150 ceph-mon[35328]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 -- 172.21.15.134:0/2172704883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000fb1e0 msgr2=0x7fb9000fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 --2- 172.21.15.134:0/2172704883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000fb1e0 0x7fb9000fb630 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc004940 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 -- 172.21.15.134:0/2172704883 shutdown_connections 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 --2- 172.21.15.134:0/2172704883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9000fb1e0 0x7fb9000fb630 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 --2- 172.21.15.134:0/2172704883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb900102c70 0x7fb900103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 -- 172.21.15.134:0/2172704883 >> 172.21.15.134:0/2172704883 conn(0x7fb9000711a0 msgr2=0x7fb9000715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 -- 172.21.15.134:0/2172704883 shutdown_connections 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.424+0000 7fb907608700 1 -- 172.21.15.134:0/2172704883 wait complete. 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 Processor -- start 2024-01-19T06:23:47.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 -- start start 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 0x7fb900113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001141e0 con 0x7fb900102c70 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb907608700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90010d4d0 con 0x7fb9000fb1e0 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47608/0 (socket says 172.21.15.134:47608) 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 -- 172.21.15.134:0/4052561505 learned_addr learned my addr 172.21.15.134:0/4052561505 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb9053a4700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 0x7fb900113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 -- 172.21.15.134:0/4052561505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 msgr2=0x7fb900113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 0x7fb900113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 -- 172.21.15.134:0/4052561505 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc005040 con 0x7fb900102c70 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb9053a4700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 0x7fb900113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.425+0000 7fb904ba3700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b3b0 tx=0x7fb8fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:47.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.426+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8fc00e040 con 0x7fb900102c70 2024-01-19T06:23:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.426+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8fc00bc50 con 0x7fb900102c70 2024-01-19T06:23:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.426+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8fc013420 con 0x7fb900102c70 2024-01-19T06:23:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.426+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90010d700 con 0x7fb900102c70 2024-01-19T06:23:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.426+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90010dc40 con 0x7fb900102c70 2024-01-19T06:23:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.427+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900105050 con 0x7fb900102c70 2024-01-19T06:23:47.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.427+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fb8fc01c430 con 0x7fb900102c70 2024-01-19T06:23:47.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.428+0000 7fb8f5ffb700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 0x7fb8ec06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:47.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.428+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb8fc087790 con 0x7fb900102c70 2024-01-19T06:23:47.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.428+0000 7fb9053a4700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 0x7fb8ec06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:47.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.428+0000 7fb9053a4700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 0x7fb8ec06b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007fd0 tx=0x7fb8f0009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:47.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.432+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb8fc057b80 con 0x7fb900102c70 2024-01-19T06:23:47.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.616+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb900072cc0 con 0x7fb900102c70 2024-01-19T06:23:47.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.616+0000 7fb8f5ffb700 1 -- 172.21.15.134:0/4052561505 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11123 (secure 0 0 0) 0x7fb8fc057710 con 0x7fb900102c70 2024-01-19T06:23:47.617 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:23:47.617 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":40,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","created":"2024-01-19T06:18:02.238207+0000","modified":"2024-01-19T06:23:28.402456+0000","last_up_change":"2024-01-19T06:23:27.392310+0000","last_in_change":"2024-01-19T06:23:08.932614+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T06:22:14.846038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6866d2e8-d629-4fd8-ab59-37109281459b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6803","nonce":3497975570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6805","nonce":3497975570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497975570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6807","nonce":3497975570}]},"public_addr":"172.21.15.134:6803/3497975570","cluster_addr":"172.21.15.134:6805/3497975570","heartbeat_back_addr":"172.21.15.134:6809/3497975570","heartbeat_front_addr":"172.21.15.134:6807/3497975570","state":["exists","up"]},{"osd":1,"uuid":"c8d35ff7-48c7-49f5-afa6-e43f5d10212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6811","nonce":743565436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6813","nonce":743565436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6817","nonce":743565436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6815","nonce":743565436}]},"public_addr":"172.21.15.134:6811/743565436","cluster_addr":"172.21.15.134:6813/743565436","heartbeat_back_addr":"172.21.15.134:6817/743565436","heartbeat_front_addr":"172.21.15.134:6815/743565436","state":["exists","up"]},{"osd":2,"uuid":"b2df8d4e-2a3c-479a-8033-2b777cf50888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6819","nonce":3103353719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6821","nonce":3103353719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6825","nonce":3103353719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6823","nonce":3103353719}]},"public_addr":"172.21.15.134:6819/3103353719","cluster_addr":"172.21.15.134:6821/3103353719","heartbeat_back_addr":"172.21.15.134:6825/3103353719","heartbeat_front_addr":"172.21.15.134:6823/3103353719","state":["exists","up"]},{"osd":3,"uuid":"b66525f9-4493-4708-bbf7-e76de91fd5b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6801","nonce":4178102778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6803","nonce":4178102778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6807","nonce":4178102778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6805","nonce":4178102778}]},"public_addr":"172.21.15.150:6801/4178102778","cluster_addr":"172.21.15.150:6803/4178102778","heartbeat_back_addr":"172.21.15.150:6807/4178102778","heartbeat_front_addr":"172.21.15.150:6805/4178102778","state":["exists","up"]},{"osd":4,"uuid":"6efffbf8-04e9-4b01-8723-a6837b2bdb63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6809","nonce":996513041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6811","nonce":996513041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6815","nonce":996513041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6813","nonce":996513041}]},"public_addr":"172.21.15.150:6809/996513041","cluster_addr":"172.21.15.150:6811/996513041","heartbeat_back_addr":"172.21.15.150:6815/996513041","heartbeat_front_addr":"172.21.15.150:6813/996513041","state":["exists","up"]},{"osd":5,"uuid":"8e42646f-77fc-418a-9339-79b1130d4906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6817","nonce":2546848737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6819","nonce":2546848737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6823","nonce":2546848737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6821","nonce":2546848737}]},"public_addr":"172.21.15.150:6817/2546848737","cluster_addr":"172.21.15.150:6819/2546848737","heartbeat_back_addr":"172.21.15.150:6823/2546848737","heartbeat_front_addr":"172.21.15.150:6821/2546848737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:16.885118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:45.420119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:12.603236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:35.079173+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:00.172610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:26.088382+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6801/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3213922814":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/2127635670":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1113248930":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6801/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/2486479390":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/1811685245":"2024-01-20T06:18:33.202902+0000","172.21.15.134:6800/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1277951871":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1593458977":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/3378221656":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/620851043":"2024-01-20T06:18:33.202902+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 msgr2=0x7fb8ec06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 0x7fb8ec06b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0007fd0 tx=0x7fb8f0009040 comp rx=0 tx=0).stop 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 msgr2=0x7fb900113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b3b0 tx=0x7fb8fc009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 shutdown_connections 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fb8ec068f60 0x7fb8ec06b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9000fb1e0 0x7fb900113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 --2- 172.21.15.134:0/4052561505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb900102c70 0x7fb900113b90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 >> 172.21.15.134:0/4052561505 conn(0x7fb9000711a0 msgr2=0x7fb9000fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 shutdown_connections 2024-01-19T06:23:47.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:47.619+0000 7fb907608700 1 -- 172.21.15.134:0/4052561505 wait complete. 2024-01-19T06:23:48.305 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:48 smithi134 ceph-mon[28283]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:48.305 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:48 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/4052561505' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:48.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-01-19T06:23:48.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-01-19T06:23:48.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-01-19T06:23:48.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-01-19T06:23:48.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-01-19T06:23:48.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-01-19T06:23:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:48 smithi150 ceph-mon[35328]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:48 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/4052561505' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:23:48.653 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:48.659 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:50.277 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:50 smithi134 ceph-mon[28283]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:50.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:50 smithi150 ceph-mon[35328]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:50.896 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:50.896 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:50.903 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:50.905 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 -- 172.21.15.134:0/2629443894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4102c70 msgr2=0x7fc9d4103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 --2- 172.21.15.134:0/2629443894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4102c70 0x7fc9d4103040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0004940 tx=0x7fc9d0009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 -- 172.21.15.134:0/2629443894 shutdown_connections 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 --2- 172.21.15.134:0/2629443894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d40fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 --2- 172.21.15.134:0/2629443894 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d4102c70 0x7fc9d4103040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 -- 172.21.15.134:0/2629443894 >> 172.21.15.134:0/2629443894 conn(0x7fc9d40711a0 msgr2=0x7fc9d40715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 -- 172.21.15.134:0/2629443894 shutdown_connections 2024-01-19T06:23:50.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 -- 172.21.15.134:0/2629443894 wait complete. 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.919+0000 7fc9dbf52700 1 Processor -- start 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9dbf52700 1 -- start start 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9dbf52700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9dbf52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4102c70 0x7fc9d4113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9dbf52700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d4114100 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9dbf52700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d410d430 con 0x7fc9d4102c70 2024-01-19T06:23:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d9cee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:50.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d9cee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47620/0 (socket says 172.21.15.134:47620) 2024-01-19T06:23:50.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d9cee700 1 -- 172.21.15.134:0/3071751045 learned_addr learned my addr 172.21.15.134:0/3071751045 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:50.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d94ed700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4102c70 0x7fc9d4113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d9cee700 1 -- 172.21.15.134:0/3071751045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4102c70 msgr2=0x7fc9d4113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.920+0000 7fc9d9cee700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4102c70 0x7fc9d4113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9d9cee700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d0005040 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9d9cee700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009360 tx=0x7fc9d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9d000e040 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9d0013420 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9dbf52700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d410d6b0 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9dbf52700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d410dbf0 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.921+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9d001c610 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.923+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fc9d001c880 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.923+0000 7fc9dbf52700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9b8001000 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.923+0000 7fc9ca7fc700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 0x7fc9c006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:50.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.923+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc9d0087b80 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.924+0000 7fc9d94ed700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 0x7fc9c006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.924+0000 7fc9ca7fc700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 0x7fc9c0071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.924+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9c0071700 con 0x7fc9c006ec40 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.924+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9d008d1e0 con 0x7fc9d40fb1e0 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.924+0000 7fc9da4ef700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 0x7fc9c0071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:50.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.925+0000 7fc9d94ed700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 0x7fc9c006b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9d410ee10 tx=0x7fc9c400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:50.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.925+0000 7fc9da4ef700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 0x7fc9c0071050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:50.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.926+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== osd.2 v2:172.21.15.134:6818/3103353719 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc9c0071700 con 0x7fc9c006ec40 2024-01-19T06:23:50.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.951+0000 7fc9dbf52700 1 -- 172.21.15.134:0/3071751045 --> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc9b8002d60 con 0x7fc9c006ec40 2024-01-19T06:23:50.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.951+0000 7fc9ca7fc700 1 -- 172.21.15.134:0/3071751045 <== osd.2 v2:172.21.15.134:6818/3103353719 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc9b8002d60 con 0x7fc9c006ec40 2024-01-19T06:23:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 msgr2=0x7fc9c0071050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 0x7fc9c0071050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 msgr2=0x7fc9c006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 0x7fc9c006b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9d410ee10 tx=0x7fc9c400b040 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 msgr2=0x7fc9d4113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009360 tx=0x7fc9d0009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 shutdown_connections 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc9c0068f70 0x7fc9c006b420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9d40fb1e0 0x7fc9d4113570 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.134:6818/3103353719,v1:172.21.15.134:6819/3103353719] conn(0x7fc9c006ec40 0x7fc9c0071050 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 --2- 172.21.15.134:0/3071751045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4102c70 0x7fc9d4113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 >> 172.21.15.134:0/3071751045 conn(0x7fc9d40711a0 msgr2=0x7fc9d40fa0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 shutdown_connections 2024-01-19T06:23:50.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:50.952+0000 7fc9b7fff700 1 -- 172.21.15.134:0/3071751045 wait complete. 2024-01-19T06:23:52.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:52 smithi134 ceph-mon[28283]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:52.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:52 smithi150 ceph-mon[35328]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:23:54.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:54 smithi134 ceph-mon[28283]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:54.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:54 smithi150 ceph-mon[35328]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 -- 172.21.15.134:0/2439749343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 msgr2=0x7fa7400fe010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 --2- 172.21.15.134:0/2439749343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa7400fe010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa730004940 tx=0x7fa730009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 -- 172.21.15.134:0/2439749343 shutdown_connections 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 --2- 172.21.15.134:0/2439749343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 --2- 172.21.15.134:0/2439749343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa7400fe010 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.444+0000 7fa744d91700 1 -- 172.21.15.134:0/2439749343 >> 172.21.15.134:0/2439749343 conn(0x7fa7400f76c0 msgr2=0x7fa7400f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 -- 172.21.15.134:0/2439749343 shutdown_connections 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 -- 172.21.15.134:0/2439749343 wait complete. 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 Processor -- start 2024-01-19T06:23:55.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 -- start start 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740114220 con 0x7fa7400fdc40 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa744d91700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74010d570 con 0x7fa7400fe5e0 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa73e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa73e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47640/0 (socket says 172.21.15.134:47640) 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa73e59c700 1 -- 172.21.15.134:0/270195681 learned_addr learned my addr 172.21.15.134:0/270195681 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.445+0000 7fa73dd9b700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa73e59c700 1 -- 172.21.15.134:0/270195681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 msgr2=0x7fa740113bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa73e59c700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740113bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa73e59c700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa730005040 con 0x7fa7400fdc40 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa73dd9b700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740113bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T06:23:55.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa73e59c700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa730009150 tx=0x7fa730009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73000e040 con 0x7fa7400fdc40 2024-01-19T06:23:55.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa730013420 con 0x7fa7400fdc40 2024-01-19T06:23:55.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74010d7f0 con 0x7fa7400fdc40 2024-01-19T06:23:55.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73001c630 con 0x7fa7400fdc40 2024-01-19T06:23:55.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.446+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74010dd30 con 0x7fa7400fdc40 2024-01-19T06:23:55.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.447+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa74010e220 con 0x7fa7400fdc40 2024-01-19T06:23:55.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.447+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fa730013970 con 0x7fa7400fdc40 2024-01-19T06:23:55.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa72effd700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 0x7fa72806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa730087c90 con 0x7fa7400fdc40 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa73dd9b700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 0x7fa72806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa72effd700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 0x7fa728071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7280716f0 con 0x7fa72806ec30 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa73008a300 con 0x7fa7400fdc40 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa73ed9d700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 0x7fa728071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa73dd9b700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 0x7fa72806b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa734009fd0 tx=0x7fa734009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.448+0000 7fa73ed9d700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 0x7fa728071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.449+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== osd.0 v2:172.21.15.134:6802/3497975570 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa7280716f0 con 0x7fa72806ec30 2024-01-19T06:23:55.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.464+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 --> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa74005e340 con 0x7fa72806ec30 2024-01-19T06:23:55.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.464+0000 7fa72effd700 1 -- 172.21.15.134:0/270195681 <== osd.0 v2:172.21.15.134:6802/3497975570 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa74005e340 con 0x7fa72806ec30 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 msgr2=0x7fa728071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 0x7fa728071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 msgr2=0x7fa72806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 0x7fa72806b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa734009fd0 tx=0x7fa734009ee0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 msgr2=0x7fa740113690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa730009150 tx=0x7fa730009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 shutdown_connections 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6802/3497975570,v1:172.21.15.134:6803/3497975570] conn(0x7fa72806ec30 0x7fa728071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fa728068f60 0x7fa72806b410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa7400fdc40 0x7fa740113690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 --2- 172.21.15.134:0/270195681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7400fe5e0 0x7fa740113bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 >> 172.21.15.134:0/270195681 conn(0x7fa7400f76c0 msgr2=0x7fa7401014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 shutdown_connections 2024-01-19T06:23:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.465+0000 7fa744d91700 1 -- 172.21.15.134:0/270195681 wait complete. 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 -- 172.21.15.134:0/3390720994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1948101b00 msgr2=0x7f1948101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 --2- 172.21.15.134:0/3390720994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1948101b00 0x7f1948101ed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1938004940 tx=0x7f1938009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 -- 172.21.15.134:0/3390720994 shutdown_connections 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 --2- 172.21.15.134:0/3390720994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19480fbb40 0x7f19480fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 --2- 172.21.15.134:0/3390720994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1948101b00 0x7f1948101ed0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 -- 172.21.15.134:0/3390720994 >> 172.21.15.134:0/3390720994 conn(0x7f19480719b0 msgr2=0x7f1948071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 -- 172.21.15.134:0/3390720994 shutdown_connections 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 -- 172.21.15.134:0/3390720994 wait complete. 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.653+0000 7f1950199700 1 Processor -- start 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 -- start start 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1948101b00 0x7f1948113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948114200 con 0x7f19480fbb40 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194810d4f0 con 0x7f1948101b00 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47658/0 (socket says 172.21.15.134:47658) 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 -- 172.21.15.134:0/3936712869 learned_addr learned my addr 172.21.15.134:0/3936712869 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194d734700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1948101b00 0x7f1948113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 -- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1948101b00 msgr2=0x7f1948113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1948101b00 0x7f1948113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938005040 con 0x7f19480fbb40 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f194df35700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f193800b3b0 tx=0x7f1938009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193800e040 con 0x7f19480fbb40 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194810d720 con 0x7f19480fbb40 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f193800bc50 con 0x7f19480fbb40 2024-01-19T06:23:55.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1938013420 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.654+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194810dc60 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.655+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f194810e100 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f193801c430 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 0x7f193406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1938087860 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f194d734700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 0x7f193406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 0x7f1934071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19340716f0 con 0x7f193406ec30 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.656+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f193808c220 con 0x7f19480fbb40 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.657+0000 7f194d734700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 0x7f193406b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1944007fd0 tx=0x7f1944009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.657+0000 7f194e736700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 0x7f1934071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:55.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.657+0000 7f194e736700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 0x7f1934071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:55.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.658+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== osd.3 v2:172.21.15.150:6800/4178102778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f19340716f0 con 0x7f193406ec30 2024-01-19T06:23:55.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.673+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 --> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f19480734d0 con 0x7f193406ec30 2024-01-19T06:23:55.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.673+0000 7f193e7fc700 1 -- 172.21.15.134:0/3936712869 <== osd.3 v2:172.21.15.150:6800/4178102778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f19480734d0 con 0x7f193406ec30 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 msgr2=0x7f1934071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 0x7f1934071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 msgr2=0x7f193406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 0x7f193406b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1944007fd0 tx=0x7f1944009040 comp rx=0 tx=0).stop 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 msgr2=0x7f1948113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f193800b3b0 tx=0x7f1938009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 shutdown_connections 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:6800/4178102778,v1:172.21.15.150:6801/4178102778] conn(0x7f193406ec30 0x7f1934071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1934068f60 0x7f193406b410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19480fbb40 0x7f1948113630 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 --2- 172.21.15.134:0/3936712869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1948101b00 0x7f1948113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:55.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 >> 172.21.15.134:0/3936712869 conn(0x7f19480719b0 msgr2=0x7f1948065450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:55.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 shutdown_connections 2024-01-19T06:23:55.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:55.674+0000 7f1950199700 1 -- 172.21.15.134:0/3936712869 wait complete. 2024-01-19T06:23:56.235 INFO:teuthology.orchestra.run.smithi134.stdout:81604378647 2024-01-19T06:23:56.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-01-19T06:23:56.405 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:56 smithi134 ceph-mon[28283]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:56.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:56 smithi150 ceph-mon[35328]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:58.093 INFO:teuthology.orchestra.run.smithi134.stdout:38654705699 2024-01-19T06:23:58.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-01-19T06:23:58.287 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:23:58 smithi134 ceph-mon[28283]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:58.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:23:58 smithi150 ceph-mon[35328]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 -- 172.21.15.134:0/3668997771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 msgr2=0x7f48f00fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 --2- 172.21.15.134:0/3668997771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f00fe040 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48ec004940 tx=0x7f48ec009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 -- 172.21.15.134:0/3668997771 shutdown_connections 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 --2- 172.21.15.134:0/3668997771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 --2- 172.21.15.134:0/3668997771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f00fe040 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 -- 172.21.15.134:0/3668997771 >> 172.21.15.134:0/3668997771 conn(0x7f48f00f76c0 msgr2=0x7f48f00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 -- 172.21.15.134:0/3668997771 shutdown_connections 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.592+0000 7f48f7387700 1 -- 172.21.15.134:0/3668997771 wait complete. 2024-01-19T06:23:58.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 Processor -- start 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 -- start start 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f01141d0 con 0x7f48f00fdc70 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f7387700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f010d4c0 con 0x7f48f00fe610 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45542/0 (socket says 172.21.15.134:45542) 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 -- 172.21.15.134:0/991154340 learned_addr learned my addr 172.21.15.134:0/991154340 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f4922700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 -- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 msgr2=0x7f48f0113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec005040 con 0x7f48f00fdc70 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f4922700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.593+0000 7f48f5123700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00b3b0 tx=0x7f48ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.594+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48ec00e040 con 0x7f48f00fdc70 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.594+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f010d7a0 con 0x7f48f00fdc70 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.594+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48ec009af0 con 0x7f48f00fdc70 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.594+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48ec01c3f0 con 0x7f48f00fdc70 2024-01-19T06:23:58.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.594+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f010dce0 con 0x7f48f00fdc70 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.595+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48f010e180 con 0x7f48f00fdc70 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.595+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f48ec00bc50 con 0x7f48f00fdc70 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48e5ffb700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 0x7f48dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f48ec087e90 con 0x7f48f00fdc70 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48f4922700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 0x7f48dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48e5ffb700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 0x7f48dc071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:23:58.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f48dc0716f0 con 0x7f48dc06ec30 2024-01-19T06:23:58.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48ec08d490 con 0x7f48f00fdc70 2024-01-19T06:23:58.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48f4922700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 0x7f48dc06b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:58.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.596+0000 7f48f5924700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 0x7f48dc071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:23:58.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.597+0000 7f48f5924700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 0x7f48dc071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:23:58.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.597+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== osd.5 v2:172.21.15.150:6816/2546848737 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f48dc0716f0 con 0x7f48dc06ec30 2024-01-19T06:23:58.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.612+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 --> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f48f005e340 con 0x7f48dc06ec30 2024-01-19T06:23:58.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48e5ffb700 1 -- 172.21.15.134:0/991154340 <== osd.5 v2:172.21.15.150:6816/2546848737 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f48f005e340 con 0x7f48dc06ec30 2024-01-19T06:23:58.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 msgr2=0x7f48dc071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:58.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 0x7f48dc071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 msgr2=0x7f48dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 0x7f48dc06b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0008040 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 msgr2=0x7f48f0113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00b3b0 tx=0x7f48ec009e30 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 shutdown_connections 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f48dc068f60 0x7f48dc06b410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48f00fdc70 0x7f48f0113640 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:6816/2546848737,v1:172.21.15.150:6817/2546848737] conn(0x7f48dc06ec30 0x7f48dc071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 --2- 172.21.15.134:0/991154340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f00fe610 0x7f48f0113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.613+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 >> 172.21.15.134:0/991154340 conn(0x7f48f00f76c0 msgr2=0x7f48f01013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.614+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 shutdown_connections 2024-01-19T06:23:58.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:58.614+0000 7f48f7387700 1 -- 172.21.15.134:0/991154340 wait complete. 2024-01-19T06:23:59.062 INFO:teuthology.orchestra.run.smithi134.stdout:111669149714 2024-01-19T06:23:59.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-01-19T06:23:59.965 INFO:teuthology.orchestra.run.smithi134.stdout:167503724554 2024-01-19T06:23:59.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 -- 172.21.15.134:0/2371345884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 msgr2=0x7fc1c0106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 --2- 172.21.15.134:0/2371345884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c0106c50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8004940 tx=0x7fc1a8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 -- 172.21.15.134:0/2371345884 shutdown_connections 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 --2- 172.21.15.134:0/2371345884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c0106c50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 --2- 172.21.15.134:0/2371345884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 0x7fc1c00fe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 -- 172.21.15.134:0/2371345884 >> 172.21.15.134:0/2371345884 conn(0x7fc1c00f76c0 msgr2=0x7fc1c00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 -- 172.21.15.134:0/2371345884 shutdown_connections 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.687+0000 7fc1c5824700 1 -- 172.21.15.134:0/2371345884 wait complete. 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 Processor -- start 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 -- start start 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 0x7fc1c00fbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00fc5e0 con 0x7fc1c00fe610 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1c5824700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00fc750 con 0x7fc1c00fdc70 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45568/0 (socket says 172.21.15.134:45568) 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 -- 172.21.15.134:0/390957898 learned_addr learned my addr 172.21.15.134:0/390957898 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:00.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1beffd700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 0x7fc1c00fbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 -- 172.21.15.134:0/390957898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 msgr2=0x7fc1c00fbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 0x7fc1c00fbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.688+0000 7fc1be7fc700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8005040 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1be7fc700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800b3b0 tx=0x7fc1a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a800e040 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1a8004290 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c01198e0 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a801c3f0 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.689+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c0119dc0 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.690+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc1c005e300 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.690+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fc1a800bc50 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.690+0000 7fc1b77fe700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 0x7fc1ac06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc1a8087ed0 con 0x7fc1c00fe610 2024-01-19T06:24:00.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1beffd700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 0x7fc1ac06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1b77fe700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 0x7fc1ac071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1ac0716f0 con 0x7fc1ac06ec30 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1a808a520 con 0x7fc1c00fe610 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1bf7fe700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 0x7fc1ac071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1beffd700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 0x7fc1ac06b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0004990 tx=0x7fc1b0009ee0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.691+0000 7fc1bf7fe700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 0x7fc1ac071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.692+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== osd.1 v2:172.21.15.134:6810/743565436 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc1ac0716f0 con 0x7fc1ac06ec30 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.706+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 --> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc1c005e6b0 con 0x7fc1ac06ec30 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1b77fe700 1 -- 172.21.15.134:0/390957898 <== osd.1 v2:172.21.15.134:6810/743565436 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc1c005e6b0 con 0x7fc1ac06ec30 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 msgr2=0x7fc1ac071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 0x7fc1ac071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 msgr2=0x7fc1ac06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 0x7fc1ac06b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0004990 tx=0x7fc1b0009ee0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 msgr2=0x7fc1c00fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800b3b0 tx=0x7fc1a8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:00.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 shutdown_connections 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6810/743565436,v1:172.21.15.134:6811/743565436] conn(0x7fc1ac06ec30 0x7fc1ac071040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc1ac068f60 0x7fc1ac06b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c00fdc70 0x7fc1c00fbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 --2- 172.21.15.134:0/390957898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1c00fe610 0x7fc1c00fbfe0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 >> 172.21.15.134:0/390957898 conn(0x7fc1c00f76c0 msgr2=0x7fc1c01013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 shutdown_connections 2024-01-19T06:24:00.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:23:59.707+0000 7fc1c5824700 1 -- 172.21.15.134:0/390957898 wait complete. 2024-01-19T06:24:00.191 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:00 smithi134 ceph-mon[28283]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:00.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:00 smithi150 ceph-mon[35328]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:00.792 INFO:teuthology.orchestra.run.smithi134.stdout:60129542174 2024-01-19T06:24:00.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-01-19T06:24:01.278 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:01.558 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:01.559 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:01.559 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:01.560 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:02.122 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:02 smithi134 ceph-mon[28283]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:02.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:02 smithi150 ceph-mon[35328]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 -- 172.21.15.134:0/1011790010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 msgr2=0x7f67a0101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 --2- 172.21.15.134:0/1011790010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a0101ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6790004940 tx=0x7f6790009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 -- 172.21.15.134:0/1011790010 shutdown_connections 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 --2- 172.21.15.134:0/1011790010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 0x7f67a00fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 --2- 172.21.15.134:0/1011790010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a0101ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 -- 172.21.15.134:0/1011790010 >> 172.21.15.134:0/1011790010 conn(0x7f67a00719b0 msgr2=0x7f67a0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 -- 172.21.15.134:0/1011790010 shutdown_connections 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.222+0000 7f67a7982700 1 -- 172.21.15.134:0/1011790010 wait complete. 2024-01-19T06:24:02.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 Processor -- start 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 -- start start 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 0x7f67a010e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a010f1c0 con 0x7f67a0101b00 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a7982700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a010f300 con 0x7f67a00fbb40 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45584/0 (socket says 172.21.15.134:45584) 2024-01-19T06:24:02.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 -- 172.21.15.134:0/4174091175 learned_addr learned my addr 172.21.15.134:0/4174091175 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a571e700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 0x7f67a010e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 -- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 msgr2=0x7f67a010e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 0x7f67a010e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790005040 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f67a4f1d700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f679c00c830 tx=0x7f679c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.223+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f679c0106f0 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.224+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a0117660 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.224+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f679c010d00 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.224+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f679c012e90 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.224+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a0117b70 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.225+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f67a005e380 con 0x7f67a0101b00 2024-01-19T06:24:02.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f679c010850 con 0x7f67a0101b00 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 0x7f678c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f679c086a50 con 0x7f67a0101b00 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 0x7f678c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f678c071590 con 0x7f678c06ead0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.226+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f679c08b220 con 0x7f67a0101b00 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.227+0000 7f67a571e700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 0x7f678c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.227+0000 7f67a5f1f700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 0x7f678c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.227+0000 7f67a571e700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 0x7f678c06b420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f679000b3b0 tx=0x7f6790009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.227+0000 7f67a5f1f700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 0x7f678c070ee0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.228+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== osd.4 v2:172.21.15.150:6808/996513041 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f678c071590 con 0x7f678c06ead0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.243+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 --> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f67a005e7b0 con 0x7f678c06ead0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f6795ffb700 1 -- 172.21.15.134:0/4174091175 <== osd.4 v2:172.21.15.150:6808/996513041 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f67a005e7b0 con 0x7f678c06ead0 2024-01-19T06:24:02.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 msgr2=0x7f678c070ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 0x7f678c070ee0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 msgr2=0x7f678c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 0x7f678c06b420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f679000b3b0 tx=0x7f6790009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 msgr2=0x7f67a010ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f679c00c830 tx=0x7f679c004470 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 shutdown_connections 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:6808/996513041,v1:172.21.15.150:6809/996513041] conn(0x7f678c06ead0 0x7f678c070ee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f678c068f70 0x7f678c06b420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67a00fbb40 0x7f67a010e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 --2- 172.21.15.134:0/4174091175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67a0101b00 0x7f67a010ec30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.244+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 >> 172.21.15.134:0/4174091175 conn(0x7f67a00719b0 msgr2=0x7f67a00fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.245+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 shutdown_connections 2024-01-19T06:24:02.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:02.245+0000 7f67a7982700 1 -- 172.21.15.134:0/4174091175 wait complete. 2024-01-19T06:24:04.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:04 smithi134 ceph-mon[28283]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:04 smithi150 ceph-mon[35328]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:04.690 INFO:teuthology.orchestra.run.smithi134.stdout:146028888079 2024-01-19T06:24:04.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-01-19T06:24:06.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:06 smithi134 ceph-mon[28283]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:06.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:06 smithi150 ceph-mon[35328]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:07.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:07 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 -- 172.21.15.134:0/3534347163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 msgr2=0x7f215c101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 --2- 172.21.15.134:0/3534347163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c101ed0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f214c004940 tx=0x7f214c009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 -- 172.21.15.134:0/3534347163 shutdown_connections 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 --2- 172.21.15.134:0/3534347163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c0fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 --2- 172.21.15.134:0/3534347163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c101ed0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 -- 172.21.15.134:0/3534347163 >> 172.21.15.134:0/3534347163 conn(0x7f215c0f76c0 msgr2=0x7f215c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 -- 172.21.15.134:0/3534347163 shutdown_connections 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.717+0000 7f2162a39700 1 -- 172.21.15.134:0/3534347163 wait complete. 2024-01-19T06:24:07.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.718+0000 7f2162a39700 1 Processor -- start 2024-01-19T06:24:07.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.718+0000 7f2162a39700 1 -- start start 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c10e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c10f280 con 0x7f215c101b00 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c10f3c0 con 0x7f215c0fbb40 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42860/0 (socket says 172.21.15.134:42860) 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 -- 172.21.15.134:0/1627860183 learned_addr learned my addr 172.21.15.134:0/1627860183 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215bfff700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c10e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 -- 172.21.15.134:0/1627860183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 msgr2=0x7f215c10e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:07.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c10e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 -- 172.21.15.134:0/1627860183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c005040 con 0x7f215c101b00 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215bfff700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f215b7fe700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f215000b7c0 tx=0x7f2150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215000f6f0 con 0x7f215c101b00 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c1176f0 con 0x7f215c101b00 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f215000fd00 con 0x7f215c101b00 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2150011e90 con 0x7f215c101b00 2024-01-19T06:24:07.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.719+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c117c30 con 0x7f215c101b00 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.721+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c103fb0 con 0x7f215c101b00 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.721+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f2150010430 con 0x7f215c101b00 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.722+0000 7f2158ff9700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 0x7f214406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.722+0000 7f215bfff700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 0x7f214406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.722+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2150086880 con 0x7f215c101b00 2024-01-19T06:24:07.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.722+0000 7f215bfff700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 0x7f214406b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f214c004560 tx=0x7f214c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:07.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.727+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f21500531a0 con 0x7f215c101b00 2024-01-19T06:24:07.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.916+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f215c0fb2c0 con 0x7f215c101b00 2024-01-19T06:24:07.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.916+0000 7f2158ff9700 1 -- 172.21.15.134:0/1627860183 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f21500567c0 con 0x7f215c101b00 2024-01-19T06:24:07.916 INFO:teuthology.orchestra.run.smithi134.stdout:38654705701 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 msgr2=0x7f214406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 0x7f214406b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f214c004560 tx=0x7f214c009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 msgr2=0x7f215c10ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f215000b7c0 tx=0x7f2150009250 comp rx=0 tx=0).stop 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 shutdown_connections 2024-01-19T06:24:07.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f2144068f60 0x7f214406b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c0fbb40 0x7f215c10e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 --2- 172.21.15.134:0/1627860183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f215c101b00 0x7f215c10ec60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 >> 172.21.15.134:0/1627860183 conn(0x7f215c0f76c0 msgr2=0x7f215c0f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 shutdown_connections 2024-01-19T06:24:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:07.918+0000 7f2162a39700 1 -- 172.21.15.134:0/1627860183 wait complete. 2024-01-19T06:24:08.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:08 smithi134 ceph-mon[28283]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:08.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:08 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1627860183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-19T06:24:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:08 smithi150 ceph-mon[35328]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:08 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1627860183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-19T06:24:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 -- 172.21.15.134:0/2459389630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104104df0 msgr2=0x7f71041051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 --2- 172.21.15.134:0/2459389630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104104df0 0x7f71041051c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004940 tx=0x7f70f4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 -- 172.21.15.134:0/2459389630 shutdown_connections 2024-01-19T06:24:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 --2- 172.21.15.134:0/2459389630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104105790 0x7f71040fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 --2- 172.21.15.134:0/2459389630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104104df0 0x7f71041051c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 -- 172.21.15.134:0/2459389630 >> 172.21.15.134:0/2459389630 conn(0x7f71040fa960 msgr2=0x7f71040fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 -- 172.21.15.134:0/2459389630 shutdown_connections 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.284+0000 7f710a01e700 1 -- 172.21.15.134:0/2459389630 wait complete. 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 Processor -- start 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 -- start start 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104104df0 0x7f7104115850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104116470 con 0x7f7104105790 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f710a01e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410f710 con 0x7f7104104df0 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f7102ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f7102ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42880/0 (socket says 172.21.15.134:42880) 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f7102ffd700 1 -- 172.21.15.134:0/1500304218 learned_addr learned my addr 172.21.15.134:0/1500304218 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.285+0000 7f71037fe700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104104df0 0x7f7104115850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:09.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f7102ffd700 1 -- 172.21.15.134:0/1500304218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104104df0 msgr2=0x7f7104115850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f7102ffd700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104104df0 0x7f7104115850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f7102ffd700 1 -- 172.21.15.134:0/1500304218 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f4005040 con 0x7f7104105790 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f7102ffd700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f800b790 tx=0x7f70f8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f800f6f0 con 0x7f7104105790 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710410f9f0 con 0x7f7104105790 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70f800fd00 con 0x7f7104105790 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f80103a0 con 0x7f7104105790 2024-01-19T06:24:09.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.286+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710410ff30 con 0x7f7104105790 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.287+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7104102100 con 0x7f7104105790 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.288+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f70f800f850 con 0x7f7104105790 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.288+0000 7f70ebfff700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 0x7f70ec06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.288+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f70f8086820 con 0x7f7104105790 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.288+0000 7f71037fe700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 0x7f70ec06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:09.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.288+0000 7f71037fe700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 0x7f70ec06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004f90 tx=0x7f70f4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:09.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.292+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f70f8056ac0 con 0x7f7104105790 2024-01-19T06:24:09.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.477+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f71040fe580 con 0x7f7104105790 2024-01-19T06:24:09.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.478+0000 7f70ebfff700 1 -- 172.21.15.134:0/1500304218 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f70f8056650 con 0x7f7104105790 2024-01-19T06:24:09.478 INFO:teuthology.orchestra.run.smithi134.stdout:167503724556 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 msgr2=0x7f70ec06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 0x7f70ec06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004f90 tx=0x7f70f4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 msgr2=0x7f7104115d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f800b790 tx=0x7f70f8009220 comp rx=0 tx=0).stop 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 shutdown_connections 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f70ec068f60 0x7f70ec06b410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7104104df0 0x7f7104115850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 --2- 172.21.15.134:0/1500304218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7104105790 0x7f7104115d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 >> 172.21.15.134:0/1500304218 conn(0x7f71040fa960 msgr2=0x7f71040fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:09.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 shutdown_connections 2024-01-19T06:24:09.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:09.480+0000 7f710a01e700 1 -- 172.21.15.134:0/1500304218 wait complete. 2024-01-19T06:24:09.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-01-19T06:24:09.774 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:10.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:10 smithi134 ceph-mon[28283]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:10.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:10 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1500304218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-19T06:24:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:10 smithi150 ceph-mon[35328]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:10.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:10 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1500304218' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-19T06:24:10.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 -- 172.21.15.134:0/3226109661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fc3a0 msgr2=0x7fd6781009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226109661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fc3a0 0x7fd6781009c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004940 tx=0x7fd66c009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 -- 172.21.15.134:0/3226109661 shutdown_connections 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226109661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fc3a0 0x7fd6781009c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226109661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fba00 0x7fd6780fbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 -- 172.21.15.134:0/3226109661 >> 172.21.15.134:0/3226109661 conn(0x7fd6780f76c0 msgr2=0x7fd6780f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 -- 172.21.15.134:0/3226109661 shutdown_connections 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.467+0000 7fd67df07700 1 -- 172.21.15.134:0/3226109661 wait complete. 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 Processor -- start 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 -- start start 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 0x7fd67811a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67811a920 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd67df07700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678113c30 con 0x7fd6780fc3a0 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd6777fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42888/0 (socket says 172.21.15.134:42888) 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.468+0000 7fd6777fe700 1 -- 172.21.15.134:0/3226238966 learned_addr learned my addr 172.21.15.134:0/3226238966 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd676ffd700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 0x7fd67811a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd6777fe700 1 -- 172.21.15.134:0/3226238966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 msgr2=0x7fd67811a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd6777fe700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 0x7fd67811a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd6777fe700 1 -- 172.21.15.134:0/3226238966 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c005040 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd676ffd700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 0x7fd67811a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd6777fe700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c810 tx=0x7fd668004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd668012810 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd668012e20 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd678113f10 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd66800a370 con 0x7fd6780fba00 2024-01-19T06:24:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.469+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd678114450 con 0x7fd6780fba00 2024-01-19T06:24:10.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.470+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6780fe0e0 con 0x7fd6780fba00 2024-01-19T06:24:10.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.471+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fd668011780 con 0x7fd6780fba00 2024-01-19T06:24:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.471+0000 7fd65ffff700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 0x7fd66006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.471+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd668086ef0 con 0x7fd6780fba00 2024-01-19T06:24:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.471+0000 7fd676ffd700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 0x7fd66006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.471+0000 7fd676ffd700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 0x7fd66006b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004560 tx=0x7fd66c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:10.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.475+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd668056e80 con 0x7fd6780fba00 2024-01-19T06:24:10.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.661+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd6780fb2c0 con 0x7fd6780fba00 2024-01-19T06:24:10.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.661+0000 7fd65ffff700 1 -- 172.21.15.134:0/3226238966 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd668016020 con 0x7fd6780fba00 2024-01-19T06:24:10.661 INFO:teuthology.orchestra.run.smithi134.stdout:81604378650 2024-01-19T06:24:10.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 msgr2=0x7fd66006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:10.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 0x7fd66006b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004560 tx=0x7fd66c009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 msgr2=0x7fd678119d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c810 tx=0x7fd668004470 comp rx=0 tx=0).stop 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 shutdown_connections 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd660068f60 0x7fd66006b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd6780fba00 0x7fd678119d90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 --2- 172.21.15.134:0/3226238966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6780fc3a0 0x7fd67811a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.664+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 >> 172.21.15.134:0/3226238966 conn(0x7fd6780f76c0 msgr2=0x7fd67810d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.665+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 shutdown_connections 2024-01-19T06:24:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:10.665+0000 7fd67df07700 1 -- 172.21.15.134:0/3226238966 wait complete. 2024-01-19T06:24:11.012 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:11.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-01-19T06:24:11.037 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:11.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:11 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3226238966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-19T06:24:11.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:11 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3226238966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-19T06:24:12.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:12 smithi134 ceph-mon[28283]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:12.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:12 smithi150 ceph-mon[35328]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:12.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378650 for osd.2 2024-01-19T06:24:12.904 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 -- 172.21.15.134:0/326649001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 msgr2=0x7f77e8106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 --2- 172.21.15.134:0/326649001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8106bf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f77d8004940 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 -- 172.21.15.134:0/326649001 shutdown_connections 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 --2- 172.21.15.134:0/326649001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8106bf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 --2- 172.21.15.134:0/326649001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 0x7f77e80fdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 -- 172.21.15.134:0/326649001 >> 172.21.15.134:0/326649001 conn(0x7f77e80f76c0 msgr2=0x7f77e80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:13.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 -- 172.21.15.134:0/326649001 shutdown_connections 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.313+0000 7f77eff3c700 1 -- 172.21.15.134:0/326649001 wait complete. 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 Processor -- start 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 -- start start 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 0x7f77e81136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8114240 con 0x7f77e80fe5b0 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77eff3c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e810d530 con 0x7f77e80fdc10 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77ed4d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77ed4d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42896/0 (socket says 172.21.15.134:42896) 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77ed4d7700 1 -- 172.21.15.134:0/287934803 learned_addr learned my addr 172.21.15.134:0/287934803 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.314+0000 7f77edcd8700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 0x7f77e81136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77ed4d7700 1 -- 172.21.15.134:0/287934803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 msgr2=0x7f77e81136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77ed4d7700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 0x7f77e81136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77ed4d7700 1 -- 172.21.15.134:0/287934803 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8005040 con 0x7f77e80fe5b0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77ed4d7700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b3b0 tx=0x7f77d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d800e040 con 0x7f77e80fe5b0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e810d7b0 con 0x7f77e80fe5b0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77d8004290 con 0x7f77e80fe5b0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d801c3f0 con 0x7f77e80fe5b0 2024-01-19T06:24:13.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.315+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e810dcf0 con 0x7f77e80fe5b0 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.316+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e8113340 con 0x7f77e80fe5b0 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.316+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f77d8013820 con 0x7f77e80fe5b0 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.317+0000 7f77de7fc700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 0x7f77d406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.317+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f77d8087d40 con 0x7f77e80fe5b0 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.317+0000 7f77edcd8700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 0x7f77d406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.317+0000 7f77edcd8700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 0x7f77d406b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009bf0 tx=0x7f77e40091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.321+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f77d8057fe0 con 0x7f77e80fe5b0 2024-01-19T06:24:13.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.511+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f77e804bb80 con 0x7f77e80fe5b0 2024-01-19T06:24:13.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.511+0000 7f77de7fc700 1 -- 172.21.15.134:0/287934803 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77d8057b70 con 0x7f77e80fe5b0 2024-01-19T06:24:13.512 INFO:teuthology.orchestra.run.smithi134.stdout:60129542176 2024-01-19T06:24:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 msgr2=0x7f77d406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 0x7f77d406b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009bf0 tx=0x7f77e40091a0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 msgr2=0x7f77e8113bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b3b0 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:13.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 shutdown_connections 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77d4068f60 0x7f77d406b410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e80fdc10 0x7f77e81136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 --2- 172.21.15.134:0/287934803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e80fe5b0 0x7f77e8113bf0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 >> 172.21.15.134:0/287934803 conn(0x7f77e80f76c0 msgr2=0x7f77e8101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 shutdown_connections 2024-01-19T06:24:13.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.514+0000 7f77eff3c700 1 -- 172.21.15.134:0/287934803 wait complete. 2024-01-19T06:24:13.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 -- 172.21.15.134:0/461259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 msgr2=0x7f52500fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 --2- 172.21.15.134:0/461259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f52500fb630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5240004940 tx=0x7f5240009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 -- 172.21.15.134:0/461259687 shutdown_connections 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 --2- 172.21.15.134:0/461259687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f52500fb630 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 --2- 172.21.15.134:0/461259687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 0x7f5250103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 -- 172.21.15.134:0/461259687 >> 172.21.15.134:0/461259687 conn(0x7f52500711a0 msgr2=0x7f52500715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 -- 172.21.15.134:0/461259687 shutdown_connections 2024-01-19T06:24:13.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.729+0000 7f5257872700 1 -- 172.21.15.134:0/461259687 wait complete. 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 Processor -- start 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 -- start start 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 0x7f5250113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52501141e0 con 0x7f52500fb1e0 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5257872700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525010d4d0 con 0x7f5250102c70 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f525560e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f525560e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42914/0 (socket says 172.21.15.134:42914) 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f525560e700 1 -- 172.21.15.134:0/1788296970 learned_addr learned my addr 172.21.15.134:0/1788296970 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.730+0000 7f5254e0d700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 0x7f5250113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f525560e700 1 -- 172.21.15.134:0/1788296970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 msgr2=0x7f5250113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f525560e700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 0x7f5250113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f525560e700 1 -- 172.21.15.134:0/1788296970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5240005040 con 0x7f52500fb1e0 2024-01-19T06:24:13.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f525560e700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f524c002a50 tx=0x7f524c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f524c007cc0 con 0x7f52500fb1e0 2024-01-19T06:24:13.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f524c00f040 con 0x7f52500fb1e0 2024-01-19T06:24:13.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525010d760 con 0x7f52500fb1e0 2024-01-19T06:24:13.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f524c00d9c0 con 0x7f52500fb1e0 2024-01-19T06:24:13.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.731+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525010dca0 con 0x7f52500fb1e0 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.732+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5250105050 con 0x7f52500fb1e0 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.732+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f524c00db20 con 0x7f52500fb1e0 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.733+0000 7f5245ffb700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 0x7f523c06f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.733+0000 7f5254e0d700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 0x7f523c06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.733+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f524c086710 con 0x7f52500fb1e0 2024-01-19T06:24:13.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.733+0000 7f5254e0d700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 0x7f523c06f830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f525010eeb0 tx=0x7f5240009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.737+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f524c056950 con 0x7f52500fb1e0 2024-01-19T06:24:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.928+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f525010ebf0 con 0x7f52500fb1e0 2024-01-19T06:24:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.929+0000 7f5245ffb700 1 -- 172.21.15.134:0/1788296970 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f524c0564e0 con 0x7f52500fb1e0 2024-01-19T06:24:13.929 INFO:teuthology.orchestra.run.smithi134.stdout:146028888081 2024-01-19T06:24:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 msgr2=0x7f523c06f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 0x7f523c06f830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f525010eeb0 tx=0x7f5240009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 msgr2=0x7f5250113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f524c002a50 tx=0x7f524c0046b0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 shutdown_connections 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f523c06d380 0x7f523c06f830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52500fb1e0 0x7f5250113610 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 --2- 172.21.15.134:0/1788296970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5250102c70 0x7f5250113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 >> 172.21.15.134:0/1788296970 conn(0x7f52500711a0 msgr2=0x7f52500fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 shutdown_connections 2024-01-19T06:24:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.931+0000 7f5257872700 1 -- 172.21.15.134:0/1788296970 wait complete. 2024-01-19T06:24:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/4235747660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 msgr2=0x7fc6d40fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/4235747660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d40fbfb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0004940 tx=0x7fc6d0009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/4235747660 shutdown_connections 2024-01-19T06:24:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/4235747660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d40fbfb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/4235747660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 0x7fc6d4101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/4235747660 >> 172.21.15.134:0/4235747660 conn(0x7fc6d40719b0 msgr2=0x7fc6d4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/4235747660 shutdown_connections 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/4235747660 wait complete. 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 Processor -- start 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 -- start start 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.932+0000 7fc6dc0f0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 0x7fc6d410ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6dc0f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d410f290 con 0x7fc6d40fbb40 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6dc0f0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d410f400 con 0x7fc6d4101b00 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42928/0 (socket says 172.21.15.134:42928) 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 -- 172.21.15.134:0/2275257878 learned_addr learned my addr 172.21.15.134:0/2275257878 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d968b700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 0x7fc6d410ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 -- 172.21.15.134:0/2275257878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 msgr2=0x7fc6d410ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 0x7fc6d410ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 -- 172.21.15.134:0/2275257878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d0005040 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6d9e8c700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c830 tx=0x7fc6c4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c40106f0 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c4010d00 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d4117770 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4012e90 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.933+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d4117c80 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.934+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d4103fb0 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.935+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fc6c4010850 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.936+0000 7fc6ca7fc700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 0x7fc6c006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.936+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc6c4087480 con 0x7fc6d40fbb40 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.936+0000 7fc6d968b700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 0x7fc6c006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.936+0000 7fc6d968b700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 0x7fc6c006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4065e30 tx=0x7fc6d0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:13.939+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc6c4053d30 con 0x7fc6d40fbb40 2024-01-19T06:24:14.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.132+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc6d40734d0 con 0x7fc6d40fbb40 2024-01-19T06:24:14.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:14 smithi134 ceph-mon[28283]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:14.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:14 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/287934803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-19T06:24:14.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:14 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1788296970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-19T06:24:14.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.133+0000 7fc6ca7fc700 1 -- 172.21.15.134:0/2275257878 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc6c4057350 con 0x7fc6d40fbb40 2024-01-19T06:24:14.133 INFO:teuthology.orchestra.run.smithi134.stdout:111669149718 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 msgr2=0x7fc6c006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 0x7fc6c006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4065e30 tx=0x7fc6d0009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 msgr2=0x7fc6d410e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c830 tx=0x7fc6c4004470 comp rx=0 tx=0).stop 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 shutdown_connections 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc6c0068f60 0x7fc6c006b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d40fbb40 0x7fc6d410e7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 --2- 172.21.15.134:0/2275257878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6d4101b00 0x7fc6d410ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 >> 172.21.15.134:0/2275257878 conn(0x7fc6d40719b0 msgr2=0x7fc6d40fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:14.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 shutdown_connections 2024-01-19T06:24:14.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:14.135+0000 7fc6dc0f0700 1 -- 172.21.15.134:0/2275257878 wait complete. 2024-01-19T06:24:14.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:14 smithi150 ceph-mon[35328]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:14.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:14 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/287934803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-19T06:24:14.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:14 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1788296970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-19T06:24:14.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-01-19T06:24:14.881 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:15.158 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:15 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2275257878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-19T06:24:15.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:15 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2275257878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-19T06:24:15.546 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-01-19T06:24:15.547 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:15.914 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149714 got 111669149718 for osd.3 2024-01-19T06:24:15.914 DEBUG:teuthology.parallel:result is None 2024-01-19T06:24:15.914 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-01-19T06:24:15.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-19T06:24:16.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:16 smithi134 ceph-mon[28283]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:16.179 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:16 smithi150 ceph-mon[35328]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 -- 172.21.15.134:0/4225571271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9640fbb40 msgr2=0x7fd9640fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 --2- 172.21.15.134:0/4225571271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9640fbb40 0x7fd9640fbfb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd954004940 tx=0x7fd954009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 -- 172.21.15.134:0/4225571271 shutdown_connections 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 --2- 172.21.15.134:0/4225571271 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9640fbb40 0x7fd9640fbfb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 --2- 172.21.15.134:0/4225571271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd964101b00 0x7fd964101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:17.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 -- 172.21.15.134:0/4225571271 >> 172.21.15.134:0/4225571271 conn(0x7fd9640f76c0 msgr2=0x7fd9640f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 -- 172.21.15.134:0/4225571271 shutdown_connections 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.958+0000 7fd96b34c700 1 -- 172.21.15.134:0/4225571271 wait complete. 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 Processor -- start 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 -- start start 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9640fbb40 0x7fd9641136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd964114260 con 0x7fd964101b00 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd96b34c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96410d5a0 con 0x7fd9640fbb40 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd9688e7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd9688e7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57888/0 (socket says 172.21.15.134:57888) 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd9688e7700 1 -- 172.21.15.134:0/1738448110 learned_addr learned my addr 172.21.15.134:0/1738448110 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.959+0000 7fd9690e8700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9640fbb40 0x7fd9641136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd9688e7700 1 -- 172.21.15.134:0/1738448110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9640fbb40 msgr2=0x7fd9641136d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:17.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd9688e7700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9640fbb40 0x7fd9641136d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd9688e7700 1 -- 172.21.15.134:0/1738448110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd954005040 con 0x7fd964101b00 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd9688e7700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b3b0 tx=0x7fd954009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd95400e040 con 0x7fd964101b00 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd95401c430 con 0x7fd964101b00 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96410d820 con 0x7fd964101b00 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd954013640 con 0x7fd964101b00 2024-01-19T06:24:17.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.960+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96410dd60 con 0x7fd964101b00 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.961+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964113360 con 0x7fd964101b00 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.962+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fd95401c5a0 con 0x7fd964101b00 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.962+0000 7fd959ffb700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 0x7fd95006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.962+0000 7fd9690e8700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 0x7fd95006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.962+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9540875a0 con 0x7fd964101b00 2024-01-19T06:24:17.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.963+0000 7fd9690e8700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 0x7fd95006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd95c004f40 tx=0x7fd95c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:17.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:17.967+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd9540578f0 con 0x7fd964101b00 2024-01-19T06:24:18.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.149+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd96410ec70 con 0x7fd950068f60 2024-01-19T06:24:18.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.150+0000 7fd959ffb700 1 -- 172.21.15.134:0/1738448110 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28007 (secure 0 0 0) 0x7fd96410ec70 con 0x7fd950068f60 2024-01-19T06:24:18.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:18.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 msgr2=0x7fd95006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:18.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 0x7fd95006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd95c004f40 tx=0x7fd95c00b040 comp rx=0 tx=0).stop 2024-01-19T06:24:18.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 msgr2=0x7fd964113c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd95400b3b0 tx=0x7fd954009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 shutdown_connections 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fd950068f60 0x7fd95006b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9640fbb40 0x7fd9641136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 --2- 172.21.15.134:0/1738448110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd964101b00 0x7fd964113c10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 >> 172.21.15.134:0/1738448110 conn(0x7fd9640f76c0 msgr2=0x7fd964065da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 shutdown_connections 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:18.152+0000 7fd96b34c700 1 -- 172.21.15.134:0/1738448110 wait complete. 2024-01-19T06:24:18.153 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-01-19T06:24:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:17 smithi134 ceph-mon[28283]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:17 smithi150 ceph-mon[35328]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:18.950 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-01-19T06:24:16.562280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002363"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-19T06:23:27.398498+0000","last_change":"2024-01-19T06:23:02.114307+0000","last_active":"2024-01-19T06:23:27.398498+0000","last_peered":"2024-01-19T06:23:27.398498+0000","last_clean":"2024-01-19T06:23:27.398498+0000","last_became_active":"2024-01-19T06:23:02.114009+0000","last_became_peered":"2024-01-19T06:23:02.114009+0000","last_unstale":"2024-01-19T06:23:27.398498+0000","last_undegraded":"2024-01-19T06:23:27.398498+0000","last_fullsized":"2024-01-19T06:23:27.398498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T06:22:15.465688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T06:22:15.465688+0000","last_clean_scrub_stamp":"2024-01-19T06:22:15.465688+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-20T17:27:54.445917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.62}]},{"osd":1,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":3,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.52800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":8.5380000000000003,"5min":8.5380000000000003,"15min":8.5380000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.504}]},{"osd":2,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":8.5090000000000003,"5min":8.5090000000000003,"15min":8.5090000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Jan 19 06:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.65500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:24:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35899999999999999,"15min":0.36399999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.28399999999999997}]},{"osd":1,"last update":"Fri Jan 19 06:24:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.30599999999999999}]},{"osd":3,"last update":"Fri Jan 19 06:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.435},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42599999999999999}]},{"osd":4,"last update":"Fri Jan 19 06:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 19 06:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.316},"min":{"1min":0.28399999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48099999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.318}]},{"osd":2,"last update":"Fri Jan 19 06:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jan 19 06:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33600000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Jan 19 06:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.318},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33900000000000002}]},{"osd":3,"last update":"Fri Jan 19 06:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.622},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.438}]},{"osd":4,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":6.9329999999999998,"5min":6.9329999999999998,"15min":6.9329999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-19T06:24:18.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-19T06:24:19.206 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:20.354 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:19 smithi134 ceph-mon[28283]: from='client.14414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T06:24:20.354 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:19 smithi134 ceph-mon[28283]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:19 smithi150 ceph-mon[35328]: from='client.14414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T06:24:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:19 smithi150 ceph-mon[35328]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:20.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- 172.21.15.134:0/1309520290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 msgr2=0x7f93680fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 --2- 172.21.15.134:0/1309520290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f93680fe040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9364004940 tx=0x7f9364009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- 172.21.15.134:0/1309520290 shutdown_connections 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 --2- 172.21.15.134:0/1309520290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 0x7f9368106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 --2- 172.21.15.134:0/1309520290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f93680fe040 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- 172.21.15.134:0/1309520290 >> 172.21.15.134:0/1309520290 conn(0x7f93680f76c0 msgr2=0x7f93680f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- 172.21.15.134:0/1309520290 shutdown_connections 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- 172.21.15.134:0/1309520290 wait complete. 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 Processor -- start 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- start start 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:20.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 0x7f9368113c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9368114250 con 0x7f93680fdc70 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.879+0000 7f9370424700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936810d540 con 0x7f93680fe610 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57900/0 (socket says 172.21.15.134:57900) 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 -- 172.21.15.134:0/3915966850 learned_addr learned my addr 172.21.15.134:0/3915966850 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936d9bf700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 0x7f9368113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 -- 172.21.15.134:0/3915966850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 msgr2=0x7f9368113c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 0x7f9368113c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 -- 172.21.15.134:0/3915966850 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364005040 con 0x7f93680fdc70 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f936e1c0700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f936400b3b0 tx=0x7f9364009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f936400e040 con 0x7f93680fdc70 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9364004290 con 0x7f93680fdc70 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936810d7d0 con 0x7f93680fdc70 2024-01-19T06:24:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f936401c3f0 con 0x7f93680fdc70 2024-01-19T06:24:20.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.880+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936810dd10 con 0x7f93680fdc70 2024-01-19T06:24:20.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.881+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9368104260 con 0x7f93680fdc70 2024-01-19T06:24:20.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.882+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f936400bc50 con 0x7f93680fdc70 2024-01-19T06:24:20.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.882+0000 7f935effd700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 0x7f935406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:20.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.882+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9364087de0 con 0x7f93680fdc70 2024-01-19T06:24:20.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.882+0000 7f936d9bf700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 0x7f935406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:20.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.883+0000 7f936d9bf700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 0x7f935406b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f936810ef20 tx=0x7f9358009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:20.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:20.887+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93640581d0 con 0x7f93680fdc70 2024-01-19T06:24:21.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.065+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f936804bb80 con 0x7f9354068f60 2024-01-19T06:24:21.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.067+0000 7f935effd700 1 -- 172.21.15.134:0/3915966850 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27949 (secure 0 0 0) 0x7f936804bb80 con 0x7f9354068f60 2024-01-19T06:24:21.067 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 msgr2=0x7f935406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 0x7f935406b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f936810ef20 tx=0x7f9358009040 comp rx=0 tx=0).stop 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 msgr2=0x7f9368113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f936400b3b0 tx=0x7f9364009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 shutdown_connections 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9354068f60 0x7f935406b410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93680fdc70 0x7f9368113680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:21.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 --2- 172.21.15.134:0/3915966850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93680fe610 0x7f9368113c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:21.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 >> 172.21.15.134:0/3915966850 conn(0x7f93680f76c0 msgr2=0x7f93681013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:21.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 shutdown_connections 2024-01-19T06:24:21.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:21.069+0000 7f9370424700 1 -- 172.21.15.134:0/3915966850 wait complete. 2024-01-19T06:24:21.070 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-01-19T06:24:21.777 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-01-19T06:24:20.563066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002244"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-19T06:23:27.398498+0000","last_change":"2024-01-19T06:23:02.114307+0000","last_active":"2024-01-19T06:23:27.398498+0000","last_peered":"2024-01-19T06:23:27.398498+0000","last_clean":"2024-01-19T06:23:27.398498+0000","last_became_active":"2024-01-19T06:23:02.114009+0000","last_became_peered":"2024-01-19T06:23:02.114009+0000","last_unstale":"2024-01-19T06:23:27.398498+0000","last_undegraded":"2024-01-19T06:23:27.398498+0000","last_fullsized":"2024-01-19T06:23:27.398498+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T06:22:15.465688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T06:22:15.465688+0000","last_clean_scrub_stamp":"2024-01-19T06:22:15.465688+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-20T17:27:54.445917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.62}]},{"osd":1,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.504}]},{"osd":3,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.52800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":8.5380000000000003,"5min":8.5380000000000003,"15min":8.5380000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38600000000000001}]},{"osd":2,"last update":"Fri Jan 19 06:23:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":8.5090000000000003,"5min":8.5090000000000003,"15min":8.5090000000000003},"last":0.376},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Jan 19 06:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.317},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:24:14 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33900000000000002,"15min":0.34100000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35899999999999999,"15min":0.36399999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Jan 19 06:24:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36599999999999999,"15min":0.36699999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36899999999999999,"15min":0.371},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.376}]},{"osd":3,"last update":"Fri Jan 19 06:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48099999999999998}]},{"osd":4,"last update":"Fri Jan 19 06:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.497},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 19 06:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.316},"min":{"1min":0.28399999999999997,"5min":0.218,"15min":0.218},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48099999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.379}]},{"osd":2,"last update":"Fri Jan 19 06:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.375,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48499999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.40500000000000003}]},{"osd":3,"last update":"Fri Jan 19 06:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 06:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33600000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.36099999999999999}]},{"osd":2,"last update":"Fri Jan 19 06:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.38200000000000001,"15min":0.38800000000000001},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.5,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.373,"15min":0.379},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Jan 19 06:23:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.73599999999999999}]},{"osd":4,"last update":"Fri Jan 19 06:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":6.9329999999999998,"5min":6.9329999999999998,"15min":6.9329999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-19T06:24:21.778 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-01-19T06:24:21.778 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-19T06:24:21.778 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-01-19T06:24:21.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-19T06:24:22.033 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:22.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:21 smithi134 ceph-mon[28283]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:22.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:21 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:21 smithi150 ceph-mon[35328]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:21 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:23.142 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:22 smithi134 ceph-mon[28283]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T06:24:23.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:22 smithi150 ceph-mon[35328]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T06:24:23.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 -- 172.21.15.134:0/430372927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62900fbb40 msgr2=0x7f62900fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 --2- 172.21.15.134:0/430372927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62900fbb40 0x7f62900fbfb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6284004940 tx=0x7f6284009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 -- 172.21.15.134:0/430372927 shutdown_connections 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 --2- 172.21.15.134:0/430372927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62900fbb40 0x7f62900fbfb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 --2- 172.21.15.134:0/430372927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6290101b00 0x7f6290101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 -- 172.21.15.134:0/430372927 >> 172.21.15.134:0/430372927 conn(0x7f62900719b0 msgr2=0x7f6290071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 -- 172.21.15.134:0/430372927 shutdown_connections 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.649+0000 7f629860e700 1 -- 172.21.15.134:0/430372927 wait complete. 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 Processor -- start 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 -- start start 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62900fbb40 0x7f6290112590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:23.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290113120 con 0x7f6290101b00 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f629860e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629010c450 con 0x7f62900fbb40 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57916/0 (socket says 172.21.15.134:57916) 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 -- 172.21.15.134:0/1366277604 learned_addr learned my addr 172.21.15.134:0/1366277604 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f62963aa700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62900fbb40 0x7f6290112590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 -- 172.21.15.134:0/1366277604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62900fbb40 msgr2=0x7f6290112590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62900fbb40 0x7f6290112590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 -- 172.21.15.134:0/1366277604 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284005040 con 0x7f6290101b00 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.650+0000 7f6295ba9700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6284009fd0 tx=0x7f6284009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.651+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f628400e040 con 0x7f6290101b00 2024-01-19T06:24:23.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.651+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6284013420 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.651+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629010c6d0 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.651+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f628401c630 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.651+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629010cc10 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.652+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6290103fb0 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.652+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f6284013590 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.652+0000 7f628effd700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 0x7f627806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.653+0000 7f62963aa700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 0x7f627806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.653+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6284087bf0 con 0x7f6290101b00 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.653+0000 7f62963aa700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 0x7f627806b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6288004550 tx=0x7f6288007eb0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.657+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6284057f30 con 0x7f6290101b00 2024-01-19T06:24:23.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.885+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6290103fb0 con 0x7f6290101b00 2024-01-19T06:24:23.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.885+0000 7f628effd700 1 -- 172.21.15.134:0/1366277604 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6284057ac0 con 0x7f6290101b00 2024-01-19T06:24:23.885 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:23.885 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 msgr2=0x7f627806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 0x7f627806b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6288004550 tx=0x7f6288007eb0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 msgr2=0x7f6290112ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6284009fd0 tx=0x7f6284009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 shutdown_connections 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6278068f70 0x7f627806b420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62900fbb40 0x7f6290112590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 --2- 172.21.15.134:0/1366277604 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6290101b00 0x7f6290112ad0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 >> 172.21.15.134:0/1366277604 conn(0x7f62900719b0 msgr2=0x7f62900faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 shutdown_connections 2024-01-19T06:24:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:23.887+0000 7f629860e700 1 -- 172.21.15.134:0/1366277604 wait complete. 2024-01-19T06:24:24.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:24 smithi134 ceph-mon[28283]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:24.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:24 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1366277604' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-19T06:24:24.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:24 smithi150 ceph-mon[35328]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:24.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:24 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1366277604' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-19T06:24:24.662 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-01-19T06:24:24.663 INFO:tasks.cephadm:Setup complete, yielding 2024-01-19T06:24:24.663 INFO:teuthology.run_tasks:Running task print... 2024-01-19T06:24:24.672 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-01-19T06:24:24.672 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:24.680 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:24.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-01-19T06:24:24.943 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:26.282 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:26 smithi134 ceph-mon[28283]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:26 smithi150 ceph-mon[35328]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 -- 172.21.15.134:0/1866976754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fbaf0 msgr2=0x7f614c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 --2- 172.21.15.134:0/1866976754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fbaf0 0x7f614c0fbf00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6148004940 tx=0x7f6148009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 -- 172.21.15.134:0/1866976754 shutdown_connections 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 --2- 172.21.15.134:0/1866976754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fcd00 0x7f614c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 --2- 172.21.15.134:0/1866976754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fbaf0 0x7f614c0fbf00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 -- 172.21.15.134:0/1866976754 >> 172.21.15.134:0/1866976754 conn(0x7f614c0f71d0 msgr2=0x7f614c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 -- 172.21.15.134:0/1866976754 shutdown_connections 2024-01-19T06:24:26.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.510+0000 7f6153da6700 1 -- 172.21.15.134:0/1866976754 wait complete. 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 Processor -- start 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 -- start start 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 0x7f614c10c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c10d040 con 0x7f614c0fcd00 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6153da6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c10d880 con 0x7f614c0fbaf0 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48482/0 (socket says 172.21.15.134:48482) 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 -- 172.21.15.134:0/436315761 learned_addr learned my addr 172.21.15.134:0/436315761 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151b42700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 0x7f614c10c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 -- 172.21.15.134:0/436315761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 msgr2=0x7f614c10c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 0x7f614c10c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151341700 1 -- 172.21.15.134:0/436315761 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148005040 con 0x7f614c0fcd00 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.511+0000 7f6151b42700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 0x7f614c10c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:24:26.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f6151341700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b790 tx=0x7f613c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:26.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c00f770 con 0x7f614c0fcd00 2024-01-19T06:24:26.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f613c00fd80 con 0x7f614c0fcd00 2024-01-19T06:24:26.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c10db60 con 0x7f614c0fcd00 2024-01-19T06:24:26.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613c00d310 con 0x7f614c0fcd00 2024-01-19T06:24:26.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.512+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c10e0a0 con 0x7f614c0fcd00 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.513+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c04bb80 con 0x7f614c0fcd00 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.513+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f613c010420 con 0x7f614c0fcd00 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.514+0000 7f61427fc700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 0x7f613806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.514+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f613c086a50 con 0x7f614c0fcd00 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.514+0000 7f6151b42700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 0x7f613806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.514+0000 7f6151b42700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 0x7f613806b420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f614800b3b0 tx=0x7f6148009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:26.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.518+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f613c056cf0 con 0x7f614c0fcd00 2024-01-19T06:24:26.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.704+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f614c10e470 con 0x7f614c0fcd00 2024-01-19T06:24:26.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.708+0000 7f61427fc700 1 -- 172.21.15.134:0/436315761 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f613c056880 con 0x7f614c0fcd00 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.710+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 msgr2=0x7f613806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.710+0000 7f6153da6700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 0x7f613806b420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f614800b3b0 tx=0x7f6148009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.710+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 msgr2=0x7f614c10c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.710+0000 7f6153da6700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b790 tx=0x7f613c009250 comp rx=0 tx=0).stop 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 shutdown_connections 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6138068f70 0x7f613806b420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f614c0fbaf0 0x7f614c10c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 --2- 172.21.15.134:0/436315761 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f614c0fcd00 0x7f614c10c9b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 >> 172.21.15.134:0/436315761 conn(0x7f614c0f71d0 msgr2=0x7f614c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 shutdown_connections 2024-01-19T06:24:26.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:26.711+0000 7f6153da6700 1 -- 172.21.15.134:0/436315761 wait complete. 2024-01-19T06:24:27.501 INFO:teuthology.run_tasks:Running task print... 2024-01-19T06:24:27.509 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-01-19T06:24:27.510 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:27.518 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:27.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph orch status' 2024-01-19T06:24:27.786 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:28.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:28.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/436315761' entity='client.admin' 2024-01-19T06:24:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:24:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:24:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:27 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:28.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:28.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/436315761' entity='client.admin' 2024-01-19T06:24:28.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:24:28.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:28.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:24:28.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:27 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 -- 172.21.15.134:0/3373513726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fbaf0 msgr2=0x7f86000fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 --2- 172.21.15.134:0/3373513726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fbaf0 0x7f86000fbf00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f85f4004940 tx=0x7f85f4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 -- 172.21.15.134:0/3373513726 shutdown_connections 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 --2- 172.21.15.134:0/3373513726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fcd00 0x7f86000fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 --2- 172.21.15.134:0/3373513726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fbaf0 0x7f86000fbf00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 -- 172.21.15.134:0/3373513726 >> 172.21.15.134:0/3373513726 conn(0x7f86000f71d0 msgr2=0x7f86000f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 -- 172.21.15.134:0/3373513726 shutdown_connections 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.456+0000 7f86068ca700 1 -- 172.21.15.134:0/3373513726 wait complete. 2024-01-19T06:24:29.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 Processor -- start 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 -- start start 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 0x7f8600110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600066a30 con 0x7f86000fcd00 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f86068ca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600066b70 con 0x7f86000fbaf0 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f85ff7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f85ff7fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48494/0 (socket says 172.21.15.134:48494) 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f85ff7fe700 1 -- 172.21.15.134:0/660445324 learned_addr learned my addr 172.21.15.134:0/660445324 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f85fffff700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 0x7f8600110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.457+0000 7f85ff7fe700 1 -- 172.21.15.134:0/660445324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 msgr2=0x7f8600110290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85ff7fe700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 0x7f8600110290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85ff7fe700 1 -- 172.21.15.134:0/660445324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4005040 con 0x7f86000fcd00 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85fffff700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 0x7f8600110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85ff7fe700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85f000b790 tx=0x7f85f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:29.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f000f770 con 0x7f86000fcd00 2024-01-19T06:24:29.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85f000fd80 con 0x7f86000fcd00 2024-01-19T06:24:29.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600112e70 con 0x7f86000fcd00 2024-01-19T06:24:29.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f0010e80 con 0x7f86000fcd00 2024-01-19T06:24:29.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.458+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8600113380 con 0x7f86000fcd00 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.459+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860004bb80 con 0x7f86000fcd00 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.459+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f85f00106b0 con 0x7f86000fcd00 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.460+0000 7f85fcff9700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 0x7f85e806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.460+0000 7f85fffff700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 0x7f85e806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.460+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f85f0086a60 con 0x7f86000fcd00 2024-01-19T06:24:29.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.460+0000 7f85fffff700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 0x7f85e806b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85f400b3b0 tx=0x7f85f4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:29.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.464+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f85f0053270 con 0x7f86000fcd00 2024-01-19T06:24:29.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.659+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8600101650 con 0x7f85e8068f60 2024-01-19T06:24:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.660+0000 7f85fcff9700 1 -- 172.21.15.134:0/660445324 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8600101650 con 0x7f85e8068f60 2024-01-19T06:24:29.660 INFO:teuthology.orchestra.run.smithi134.stdout:Backend: cephadm 2024-01-19T06:24:29.660 INFO:teuthology.orchestra.run.smithi134.stdout:Available: Yes 2024-01-19T06:24:29.660 INFO:teuthology.orchestra.run.smithi134.stdout:Paused: No 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 msgr2=0x7f85e806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 0x7f85e806b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f85f400b3b0 tx=0x7f85f4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 msgr2=0x7f86001127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85f000b790 tx=0x7f85f0009250 comp rx=0 tx=0).stop 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 shutdown_connections 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f85e8068f60 0x7f85e806b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86000fbaf0 0x7f8600110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 --2- 172.21.15.134:0/660445324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86000fcd00 0x7f86001127e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:29.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 >> 172.21.15.134:0/660445324 conn(0x7f86000f71d0 msgr2=0x7f86000fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:29.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 shutdown_connections 2024-01-19T06:24:29.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:29.662+0000 7f86068ca700 1 -- 172.21.15.134:0/660445324 wait complete. 2024-01-19T06:24:29.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:29 smithi134 ceph-mon[28283]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:30.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:29 smithi150 ceph-mon[35328]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:30.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph orch ps' 2024-01-19T06:24:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:31.033 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:30 smithi134 ceph-mon[28283]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:31.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:30 smithi150 ceph-mon[35328]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:32.084 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:31 smithi134 ceph-mon[28283]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:32.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:31 smithi150 ceph-mon[35328]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 -- 172.21.15.134:0/3331624149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 msgr2=0x7f53500fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 --2- 172.21.15.134:0/3331624149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f53500fd000 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5340004940 tx=0x7f5340009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 -- 172.21.15.134:0/3331624149 shutdown_connections 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 --2- 172.21.15.134:0/3331624149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 0x7f53500fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 --2- 172.21.15.134:0/3331624149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f53500fd000 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 -- 172.21.15.134:0/3331624149 >> 172.21.15.134:0/3331624149 conn(0x7f53500f82d0 msgr2=0x7f53500fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:32.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 -- 172.21.15.134:0/3331624149 shutdown_connections 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.483+0000 7f535634a700 1 -- 172.21.15.134:0/3331624149 wait complete. 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 Processor -- start 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 -- start start 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 0x7f535010dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535010e140 con 0x7f53500fcbf0 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f535634a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535010e980 con 0x7f53500fde00 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48522/0 (socket says 172.21.15.134:48522) 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 -- 172.21.15.134:0/2681479373 learned_addr learned my addr 172.21.15.134:0/2681479373 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:32.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534f7fe700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 0x7f535010dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 -- 172.21.15.134:0/2681479373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 msgr2=0x7f535010dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 0x7f535010dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.484+0000 7f534ffff700 1 -- 172.21.15.134:0/2681479373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340005040 con 0x7f53500fcbf0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f534ffff700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5340009360 tx=0x7f5340009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f534000e040 con 0x7f53500fcbf0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5340004290 con 0x7f53500fcbf0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f534001c630 con 0x7f53500fcbf0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535010ec00 con 0x7f53500fcbf0 2024-01-19T06:24:32.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.485+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535010f0f0 con 0x7f53500fcbf0 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.486+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535004bb80 con 0x7f53500fcbf0 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.486+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f5340013960 con 0x7f53500fcbf0 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.487+0000 7f534cff9700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 0x7f533806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.487+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5340087dc0 con 0x7f53500fcbf0 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.487+0000 7f534f7fe700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 0x7f533806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:32.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.487+0000 7f534f7fe700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 0x7f533806b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5344006fd0 tx=0x7f5344008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:32.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.492+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5340058060 con 0x7f53500fcbf0 2024-01-19T06:24:32.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.684+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f535010f590 con 0x7f5338068f70 2024-01-19T06:24:32.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.692+0000 7f534cff9700 1 -- 172.21.15.134:0/2681479373 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f535010f590 con 0x7f5338068f70 2024-01-19T06:24:32.692 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T06:24:32.692 INFO:teuthology.orchestra.run.smithi134.stdout:alertmanager.smithi134 smithi134 *:9093,9094 running (3m) 2m ago 4m 25.7M - 0.25.0 c8568f914cd2 09f464d8c526 2024-01-19T06:24:32.692 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi134 smithi134 running (5m) 2m ago 5m 6665k - 17.2.7-268-g6441305d c3a232f785d8 434084e15158 2024-01-19T06:24:32.692 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi150 smithi150 running (4m) 66s ago 4m 6660k - 17.2.7-268-g6441305d c3a232f785d8 1aee22edee9f 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:grafana.smithi134 smithi134 *:3000 running (3m) 2m ago 4m 80.0M - 9.4.7 954c08fa6188 d3ec6951f4ff 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi134.qrxokr smithi134 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-268-g6441305d c3a232f785d8 6f69c250d5a7 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi150.xkxrco smithi150 *:8443,9283 running (4m) 66s ago 4m 432M - 17.2.7-268-g6441305d c3a232f785d8 d6d4dd5f44c3 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi134 smithi134 running (6m) 2m ago 6m 41.5M 2048M 17.2.7-268-g6441305d c3a232f785d8 328c6cb50ef7 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi150 smithi150 running (4m) 66s ago 4m 38.3M 2048M 17.2.7-268-g6441305d c3a232f785d8 36884ce908a9 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi134 smithi134 *:9100 running (5m) 2m ago 5m 15.1M - 1.5.0 0da6a335fe13 9ff3509f6e9d 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi150 smithi150 *:9100 running (4m) 66s ago 4m 15.1M - 1.5.0 0da6a335fe13 f5edbad1d4e4 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (3m) 2m ago 3m 41.2M 4288M 17.2.7-268-g6441305d c3a232f785d8 c42a610a3422 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi134 running (2m) 2m ago 2m 43.3M 4288M 17.2.7-268-g6441305d c3a232f785d8 980268e5252b 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi134 running (2m) 2m ago 2m 38.5M 4288M 17.2.7-268-g6441305d c3a232f785d8 cbc144916de2 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi150 running (2m) 66s ago 2m 38.6M 5312M 17.2.7-268-g6441305d c3a232f785d8 799be7524f32 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 smithi150 running (97s) 66s ago 97s 39.0M 5312M 17.2.7-268-g6441305d c3a232f785d8 4e0d71b857f1 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 smithi150 running (71s) 66s ago 71s 9.97M 5312M 17.2.7-268-g6441305d c3a232f785d8 175a03fc551b 2024-01-19T06:24:32.693 INFO:teuthology.orchestra.run.smithi134.stdout:prometheus.smithi134 smithi134 *:9095 running (3m) 2m ago 4m 37.9M - 2.43.0 a07b618ecd1d e5681937937a 2024-01-19T06:24:32.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 msgr2=0x7f533806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 0x7f533806b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5344006fd0 tx=0x7f5344008040 comp rx=0 tx=0).stop 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 msgr2=0x7f535010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5340009360 tx=0x7f5340009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 shutdown_connections 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5338068f70 0x7f533806b420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53500fcbf0 0x7f535010d570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 --2- 172.21.15.134:0/2681479373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53500fde00 0x7f535010dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.694+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 >> 172.21.15.134:0/2681479373 conn(0x7f53500f82d0 msgr2=0x7f5350101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.695+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 shutdown_connections 2024-01-19T06:24:32.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:32.695+0000 7f535634a700 1 -- 172.21.15.134:0/2681479373 wait complete. 2024-01-19T06:24:33.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph orch ls' 2024-01-19T06:24:33.749 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:34.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:33 smithi134 ceph-mon[28283]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:34.257 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:33 smithi134 ceph-mon[28283]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:33 smithi150 ceph-mon[35328]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:33 smithi150 ceph-mon[35328]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:35.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 -- 172.21.15.134:0/857716386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fcd00 msgr2=0x7f07240fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 --2- 172.21.15.134:0/857716386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fcd00 0x7f07240fd150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0718004940 tx=0x7f0718009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 -- 172.21.15.134:0/857716386 shutdown_connections 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 --2- 172.21.15.134:0/857716386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fcd00 0x7f07240fd150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 --2- 172.21.15.134:0/857716386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fbaf0 0x7f07240fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 -- 172.21.15.134:0/857716386 >> 172.21.15.134:0/857716386 conn(0x7f07240f71d0 msgr2=0x7f07240f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 -- 172.21.15.134:0/857716386 shutdown_connections 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.514+0000 7f0729770700 1 -- 172.21.15.134:0/857716386 wait complete. 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 Processor -- start 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 -- start start 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fcd00 0x7f072410c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:35.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072410d010 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0729770700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072410d850 con 0x7f07240fcd00 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48550/0 (socket says 172.21.15.134:48550) 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 -- 172.21.15.134:0/1870315942 learned_addr learned my addr 172.21.15.134:0/1870315942 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f07227fc700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fcd00 0x7f072410c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 -- 172.21.15.134:0/1870315942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fcd00 msgr2=0x7f072410c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fcd00 0x7f072410c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.515+0000 7f0722ffd700 1 -- 172.21.15.134:0/1870315942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718005040 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f0722ffd700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f071400b790 tx=0x7f0714009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f071400f6f0 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f071400fd00 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072410db30 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07140103a0 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.516+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072410e070 con 0x7f07240fbaf0 2024-01-19T06:24:35.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.517+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072404bb80 con 0x7f07240fbaf0 2024-01-19T06:24:35.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.517+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f071400f850 con 0x7f07240fbaf0 2024-01-19T06:24:35.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.517+0000 7f070b7fe700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 0x7f070c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:35.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.518+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0714086900 con 0x7f07240fbaf0 2024-01-19T06:24:35.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.518+0000 7f07227fc700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 0x7f070c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:35.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.518+0000 7f07227fc700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 0x7f070c06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f071800b3b0 tx=0x7f0718009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:35.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.522+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0714056ac0 con 0x7f07240fbaf0 2024-01-19T06:24:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.714+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f072410e520 con 0x7f070c068f60 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.718+0000 7f070b7fe700 1 -- 172.21.15.134:0/1870315942 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f072410e520 con 0x7f070c068f60 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:crash 2/2 2m ago 5m * 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:mgr 2/2 2m ago 5m count:2 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:mon 2/2 2m ago 4m smithi134:172.21.15.134=smithi134;smithi150:172.21.15.150=smithi150;count:2 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-01-19T06:24:35.717 INFO:teuthology.orchestra.run.smithi134.stdout:osd 6 2m ago - 2024-01-19T06:24:35.718 INFO:teuthology.orchestra.run.smithi134.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 msgr2=0x7f070c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 0x7f070c06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f071800b3b0 tx=0x7f0718009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 msgr2=0x7f072410c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f071400b790 tx=0x7f0714009220 comp rx=0 tx=0).stop 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 shutdown_connections 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f070c068f60 0x7f070c06b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07240fbaf0 0x7f072410c440 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 --2- 172.21.15.134:0/1870315942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07240fcd00 0x7f072410c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 >> 172.21.15.134:0/1870315942 conn(0x7f07240f71d0 msgr2=0x7f07240fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 shutdown_connections 2024-01-19T06:24:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:35.720+0000 7f0729770700 1 -- 172.21.15.134:0/1870315942 wait complete. 2024-01-19T06:24:36.302 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:36 smithi134 ceph-mon[28283]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:36.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph orch host ls' 2024-01-19T06:24:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:36 smithi150 ceph-mon[35328]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:36.583 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:37.375 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:37 smithi134 ceph-mon[28283]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:37.375 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:37 smithi150 ceph-mon[35328]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:38.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 -- 172.21.15.134:0/316995392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fec50 msgr2=0x7fc4240fad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 --2- 172.21.15.134:0/316995392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fec50 0x7fc4240fad00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc410004940 tx=0x7fc410009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 -- 172.21.15.134:0/316995392 shutdown_connections 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 --2- 172.21.15.134:0/316995392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fec50 0x7fc4240fad00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 --2- 172.21.15.134:0/316995392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fe270 0x7fc4240fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 -- 172.21.15.134:0/316995392 >> 172.21.15.134:0/316995392 conn(0x7fc4240f6950 msgr2=0x7fc4240f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 -- 172.21.15.134:0/316995392 shutdown_connections 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 -- 172.21.15.134:0/316995392 wait complete. 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.195+0000 7fc42ad90700 1 Processor -- start 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc42ad90700 1 -- start start 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc42ad90700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc42ad90700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fbf30 0x7fc4240fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:38.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc42ad90700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240fc930 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc42ad90700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240fca70 con 0x7fc4240fbf30 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44426/0 (socket says 172.21.15.134:44426) 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 -- 172.21.15.134:0/2033281716 learned_addr learned my addr 172.21.15.134:0/2033281716 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc423fff700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fbf30 0x7fc4240fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 -- 172.21.15.134:0/2033281716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fbf30 msgr2=0x7fc4240fc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fbf30 0x7fc4240fc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 -- 172.21.15.134:0/2033281716 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410005040 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.196+0000 7fc428b2c700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc418002a80 tx=0x7fc418004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.197+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc418007cc0 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.197+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc41800f040 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.197+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240fcd50 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.197+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc41800d9c0 con 0x7fc4240fe270 2024-01-19T06:24:38.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.197+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc424115980 con 0x7fc4240fe270 2024-01-19T06:24:38.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.198+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc424107600 con 0x7fc4240fe270 2024-01-19T06:24:38.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.198+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fc41800db20 con 0x7fc4240fe270 2024-01-19T06:24:38.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.198+0000 7fc4217fa700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 0x7fc41406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:38.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.199+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4180866c0 con 0x7fc4240fe270 2024-01-19T06:24:38.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.199+0000 7fc423fff700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 0x7fc41406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:38.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.199+0000 7fc423fff700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 0x7fc41406b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc410004f90 tx=0x7fc410009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:38.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.203+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc418056960 con 0x7fc4240fe270 2024-01-19T06:24:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.403+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4240fedc0 con 0x7fc414068f60 2024-01-19T06:24:38.403 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:38 smithi134 ceph-mon[28283]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:38.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.404+0000 7fc4217fa700 1 -- 172.21.15.134:0/2033281716 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fc4240fedc0 con 0x7fc414068f60 2024-01-19T06:24:38.404 INFO:teuthology.orchestra.run.smithi134.stdout:HOST ADDR LABELS STATUS 2024-01-19T06:24:38.405 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 172.21.15.134 2024-01-19T06:24:38.405 INFO:teuthology.orchestra.run.smithi134.stdout:smithi150 172.21.15.150 2024-01-19T06:24:38.405 INFO:teuthology.orchestra.run.smithi134.stdout:2 hosts in cluster 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 msgr2=0x7fc41406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 0x7fc41406b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc410004f90 tx=0x7fc410009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 msgr2=0x7fc4240fdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc418002a80 tx=0x7fc418004600 comp rx=0 tx=0).stop 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 shutdown_connections 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fc414068f60 0x7fc41406b410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4240fe270 0x7fc4240fdab0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 --2- 172.21.15.134:0/2033281716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4240fbf30 0x7fc4240fc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 >> 172.21.15.134:0/2033281716 conn(0x7fc4240f6950 msgr2=0x7fc4240f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 shutdown_connections 2024-01-19T06:24:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:38.407+0000 7fc42ad90700 1 -- 172.21.15.134:0/2033281716 wait complete. 2024-01-19T06:24:38.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:38 smithi150 ceph-mon[35328]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:39.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph orch device ls' 2024-01-19T06:24:39.257 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:40.373 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:40 smithi134 ceph-mon[28283]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:40.373 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:40 smithi134 ceph-mon[28283]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:40.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:40 smithi150 ceph-mon[35328]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:40.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:40 smithi150 ceph-mon[35328]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:40.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 -- 172.21.15.134:0/224512568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 msgr2=0x7f115c0fcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:40.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 --2- 172.21.15.134:0/224512568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c0fcf00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1148004940 tx=0x7f1148009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:40.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 -- 172.21.15.134:0/224512568 shutdown_connections 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 --2- 172.21.15.134:0/224512568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c0fcf00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 --2- 172.21.15.134:0/224512568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c0fc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 -- 172.21.15.134:0/224512568 >> 172.21.15.134:0/224512568 conn(0x7f115c0f69d0 msgr2=0x7f115c0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 -- 172.21.15.134:0/224512568 shutdown_connections 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.943+0000 7f1162c1a700 1 -- 172.21.15.134:0/224512568 wait complete. 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 Processor -- start 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 -- start start 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c110520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0fbbe0 con 0x7f115c0fca90 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f1162c1a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0fbd20 con 0x7f115c0fc140 2024-01-19T06:24:40.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44446/0 (socket says 172.21.15.134:44446) 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 -- 172.21.15.134:0/4195830012 learned_addr learned my addr 172.21.15.134:0/4195830012 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f11609b6700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c110520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 -- 172.21.15.134:0/4195830012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 msgr2=0x7f115c110520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c110520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.944+0000 7f115bfff700 1 -- 172.21.15.134:0/4195830012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148005040 con 0x7f115c0fca90 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f11609b6700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c110520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f115bfff700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1148009390 tx=0x7f1148009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f114800e040 con 0x7f115c0fca90 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f114800bcd0 con 0x7f115c0fca90 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c113070 con 0x7f115c0fca90 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f114801c630 con 0x7f115c0fca90 2024-01-19T06:24:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.945+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c1135b0 con 0x7f115c0fca90 2024-01-19T06:24:40.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.946+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c1064f0 con 0x7f115c0fca90 2024-01-19T06:24:40.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.946+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f1148013960 con 0x7f115c0fca90 2024-01-19T06:24:40.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.947+0000 7f11597fa700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 0x7f114c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:40.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.947+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1148087ec0 con 0x7f115c0fca90 2024-01-19T06:24:40.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.947+0000 7f11609b6700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 0x7f114c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:40.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.947+0000 7f11609b6700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 0x7f114c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11500041d0 tx=0x7f1150009040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:40.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:40.951+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f11480546d0 con 0x7f115c0fca90 2024-01-19T06:24:41.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.142+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f115c104620 con 0x7f114c068f60 2024-01-19T06:24:41.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.144+0000 7f11597fa700 1 -- 172.21.15.134:0/4195830012 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f115c104620 con 0x7f114c068f60 2024-01-19T06:24:41.144 INFO:teuthology.orchestra.run.smithi134.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-01-19T06:24:41.144 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62420002400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-01-19T06:24:41.144 INFO:teuthology.orchestra.run.smithi134.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 msgr2=0x7f114c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 0x7f114c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f11500041d0 tx=0x7f1150009040 comp rx=0 tx=0).stop 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 msgr2=0x7f115c112a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1148009390 tx=0x7f1148009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 shutdown_connections 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f114c068f60 0x7f114c06b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f115c0fc140 0x7f115c110520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 --2- 172.21.15.134:0/4195830012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f115c0fca90 0x7f115c112a70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:41.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 >> 172.21.15.134:0/4195830012 conn(0x7f115c0f69d0 msgr2=0x7f115c1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:41.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 shutdown_connections 2024-01-19T06:24:41.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:41.146+0000 7f1162c1a700 1 -- 172.21.15.134:0/4195830012 wait complete. 2024-01-19T06:24:41.832 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:41.843 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:41.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-01-19T06:24:42.104 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:42.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:42 smithi134 ceph-mon[28283]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:42.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:42 smithi150 ceph-mon[35328]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:43.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:43 smithi134 ceph-mon[28283]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:43.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:43 smithi150 ceph-mon[35328]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:43.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 -- 172.21.15.134:0/1214909277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 msgr2=0x7f39640fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:43.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 --2- 172.21.15.134:0/1214909277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f39640fd150 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3958004940 tx=0x7f3958009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 -- 172.21.15.134:0/1214909277 shutdown_connections 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 --2- 172.21.15.134:0/1214909277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f39640fd150 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 --2- 172.21.15.134:0/1214909277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 0x7f39640fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 -- 172.21.15.134:0/1214909277 >> 172.21.15.134:0/1214909277 conn(0x7f39640f71d0 msgr2=0x7f39640f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 -- 172.21.15.134:0/1214909277 shutdown_connections 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.757+0000 7f3969dcd700 1 -- 172.21.15.134:0/1214909277 wait complete. 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 Processor -- start 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 -- start start 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 0x7f3964112690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3964066890 con 0x7f39640fcd00 2024-01-19T06:24:43.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3969dcd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640669d0 con 0x7f39640fbaf0 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3962ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3962ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44468/0 (socket says 172.21.15.134:44468) 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f3962ffd700 1 -- 172.21.15.134:0/754282252 learned_addr learned my addr 172.21.15.134:0/754282252 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.758+0000 7f39637fe700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 0x7f3964112690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3962ffd700 1 -- 172.21.15.134:0/754282252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 msgr2=0x7f3964112690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3962ffd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 0x7f3964112690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3962ffd700 1 -- 172.21.15.134:0/754282252 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958005040 con 0x7f39640fcd00 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3962ffd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3958009fd0 tx=0x7f3958009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395800e040 con 0x7f39640fcd00 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3964115120 con 0x7f39640fcd00 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3958004290 con 0x7f39640fcd00 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395801c3f0 con 0x7f39640fcd00 2024-01-19T06:24:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.759+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3964115580 con 0x7f39640fcd00 2024-01-19T06:24:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.760+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396404bb80 con 0x7f39640fcd00 2024-01-19T06:24:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.760+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f395800bc50 con 0x7f39640fcd00 2024-01-19T06:24:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.761+0000 7f394bfff700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 0x7f394c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.761+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3958087e50 con 0x7f39640fcd00 2024-01-19T06:24:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.761+0000 7f39637fe700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 0x7f394c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:43.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.761+0000 7f39637fe700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 0x7f394c06b420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3964066620 tx=0x7f39540091a0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:43.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.765+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3958054660 con 0x7f39640fcd00 2024-01-19T06:24:43.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:43.981+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3964101650 con 0x7f394c068f70 2024-01-19T06:24:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:44 smithi134 ceph-mon[28283]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:44.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:44 smithi150 ceph-mon[35328]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:45.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.300+0000 7f394bfff700 1 -- 172.21.15.134:0/754282252 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3964101650 con 0x7f394c068f70 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 msgr2=0x7f394c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 0x7f394c06b420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3964066620 tx=0x7f39540091a0 comp rx=0 tx=0).stop 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 msgr2=0x7f3964114be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3958009fd0 tx=0x7f3958009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 shutdown_connections 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f394c068f70 0x7f394c06b420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39640fbaf0 0x7f3964112690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:45.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 --2- 172.21.15.134:0/754282252 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39640fcd00 0x7f3964114be0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:45.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 >> 172.21.15.134:0/754282252 conn(0x7f39640f71d0 msgr2=0x7f39640fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:45.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 shutdown_connections 2024-01-19T06:24:45.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:45.305+0000 7f3969dcd700 1 -- 172.21.15.134:0/754282252 wait complete. 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: osdmap e41: 6 total, 6 up, 6 in 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-19T06:24:45.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-mon[28283]: pgmap v168: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:45.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:24:45.276+0000 7f2e684d7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T06:24:45.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:24:45.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-19T06:24:45.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-19T06:24:45.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: osdmap e41: 6 total, 6 up, 6 in 2024-01-19T06:24:45.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-19T06:24:45.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:45 smithi150 ceph-mon[35328]: pgmap v168: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:46.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-19T06:24:46.387 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: osdmap e42: 6 total, 6 up, 6 in 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: osdmap e43: 6 total, 6 up, 6 in 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: fsmap cephfs:0 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: Saving service mds.cephfs spec with placement count:4 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:46.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.higalz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.higalz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:46.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:46 smithi134 ceph-mon[28283]: Deploying daemon mds.cephfs.smithi134.higalz on smithi134 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: osdmap e42: 6 total, 6 up, 6 in 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: osdmap e43: 6 total, 6 up, 6 in 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: fsmap cephfs:0 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: Saving service mds.cephfs spec with placement count:4 2024-01-19T06:24:46.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.higalz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.higalz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:46.684 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:46 smithi150 ceph-mon[35328]: Deploying daemon mds.cephfs.smithi134.higalz on smithi134 2024-01-19T06:24:47.413 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:47 smithi134 ceph-mon[28283]: osdmap e44: 6 total, 6 up, 6 in 2024-01-19T06:24:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:47 smithi134 ceph-mon[28283]: pgmap v172: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:47.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:47 smithi150 ceph-mon[35328]: osdmap e44: 6 total, 6 up, 6 in 2024-01-19T06:24:47.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:47 smithi150 ceph-mon[35328]: pgmap v172: 65 pgs: 8 creating+peering, 1 active+clean, 56 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:48.669 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:48 smithi134 ceph-mon[28283]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-19T06:24:48.669 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:48 smithi134 ceph-mon[28283]: osdmap e45: 6 total, 6 up, 6 in 2024-01-19T06:24:48.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:48 smithi150 ceph-mon[35328]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-19T06:24:48.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:48 smithi150 ceph-mon[35328]: osdmap e45: 6 total, 6 up, 6 in 2024-01-19T06:24:49.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- 172.21.15.134:0/2378795079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fcd00 msgr2=0x7f5ce80fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:49.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/2378795079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fcd00 0x7f5ce80fd150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8004940 tx=0x7f5cd8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:49.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- 172.21.15.134:0/2378795079 shutdown_connections 2024-01-19T06:24:49.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/2378795079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fcd00 0x7f5ce80fd150 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/2378795079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce80fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- 172.21.15.134:0/2378795079 >> 172.21.15.134:0/2378795079 conn(0x7f5ce80f7190 msgr2=0x7f5ce80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- 172.21.15.134:0/2378795079 shutdown_connections 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- 172.21.15.134:0/2378795079 wait complete. 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 Processor -- start 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.198+0000 7f5cf00d6700 1 -- start start 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cf00d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cf00d6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fcd00 0x7f5ce8112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cf00d6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8066110 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cf00d6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8066250 con 0x7f5ce80fcd00 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47082/0 (socket says 172.21.15.134:47082) 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 -- 172.21.15.134:0/3896646159 learned_addr learned my addr 172.21.15.134:0/3896646159 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5ced671700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fcd00 0x7f5ce8112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 -- 172.21.15.134:0/3896646159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fcd00 msgr2=0x7f5ce8112760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fcd00 0x7f5ce8112760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 -- 172.21.15.134:0/3896646159 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd8005040 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cede72700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400eb40 tx=0x7f5ce400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.199+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce4007cc0 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.200+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce8112d60 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.200+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ce400f040 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.200+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce400c9f0 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.200+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce81131e0 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.201+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce804bb80 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.202+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f5ce40090d0 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.202+0000 7f5cde7fc700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 0x7f5cd406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.202+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5ce40868b0 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.202+0000 7f5ced671700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 0x7f5cd406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:49.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.203+0000 7f5ced671700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 0x7f5cd406b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8000c00 tx=0x7f5cd8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:49.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.207+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5ce4052d00 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.446+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5ce8113730 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.447+0000 7f5cde7fc700 1 -- 172.21.15.134:0/3896646159 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f5ce4019450 con 0x7f5ce80fbaf0 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stdout:e2 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stdout:legacy client fscid: 1 2024-01-19T06:24:49.447 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:Filesystem 'cephfs' (1) 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:fs_name cephfs 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 2 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-01-19T06:24:45.276806+0000 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:modified 2024-01-19T06:24:45.277029+0000 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:tableserver 0 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:root 0 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:session_timeout 60 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:session_autoclose 300 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:max_file_size 1099511627776 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:required_client_features {} 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure 0 2024-01-19T06:24:49.448 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure_osd_epoch 0 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:max_mds 1 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:in 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:up {} 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:failed 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:damaged 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:stopped 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:data_pools [3] 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:metadata_pool 2 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:inline_data disabled 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:balancer 2024-01-19T06:24:49.449 INFO:teuthology.orchestra.run.smithi134.stdout:standby_count_wanted 0 2024-01-19T06:24:49.450 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:49.450 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:24:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 msgr2=0x7f5cd406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:49.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 0x7f5cd406b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8000c00 tx=0x7f5cd8009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 msgr2=0x7f5ce8110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400eb40 tx=0x7f5ce400b040 comp rx=0 tx=0).stop 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 shutdown_connections 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f5cd4068f60 0x7f5cd406b410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ce80fbaf0 0x7f5ce8110210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 --2- 172.21.15.134:0/3896646159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ce80fcd00 0x7f5ce8112760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 >> 172.21.15.134:0/3896646159 conn(0x7f5ce80f7190 msgr2=0x7f5ce80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 shutdown_connections 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:49.449+0000 7f5cf00d6700 1 -- 172.21.15.134:0/3896646159 wait complete. 2024-01-19T06:24:49.451 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 2 2024-01-19T06:24:49.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:49 smithi134 ceph-mon[28283]: pgmap v174: 65 pgs: 47 active+clean, 8 creating+peering, 10 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:49.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:49 smithi150 ceph-mon[35328]: pgmap v174: 65 pgs: 47 active+clean, 8 creating+peering, 10 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:50.340 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:50.356 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:50.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3896646159' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.xqfgpx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.xqfgpx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:50.535 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:50 smithi150 ceph-mon[35328]: Deploying daemon mds.cephfs.smithi150.xqfgpx on smithi150 2024-01-19T06:24:50.615 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3896646159' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.xqfgpx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.xqfgpx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:50.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:50.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:50 smithi134 ceph-mon[28283]: Deploying daemon mds.cephfs.smithi150.xqfgpx on smithi150 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.134:6826/3512441553,v1:172.21.15.134:6827/3512441553] up:boot 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: daemon mds.cephfs.smithi134.higalz assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: Cluster is now healthy 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: fsmap cephfs:0 1 up:standby 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.higalz"}]: dispatch 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:creating} 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: daemon mds.cephfs.smithi134.higalz is now active in filesystem cephfs as rank 0 2024-01-19T06:24:51.644 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:51 smithi150 ceph-mon[35328]: pgmap v175: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.134:6826/3512441553,v1:172.21.15.134:6827/3512441553] up:boot 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: daemon mds.cephfs.smithi134.higalz assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: Cluster is now healthy 2024-01-19T06:24:51.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: fsmap cephfs:0 1 up:standby 2024-01-19T06:24:51.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.higalz"}]: dispatch 2024-01-19T06:24:51.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:creating} 2024-01-19T06:24:51.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: daemon mds.cephfs.smithi134.higalz is now active in filesystem cephfs as rank 0 2024-01-19T06:24:51.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:51 smithi134 ceph-mon[28283]: pgmap v175: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:24:52.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 -- 172.21.15.134:0/3229130877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fcd00 msgr2=0x7f6e000fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:52.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 --2- 172.21.15.134:0/3229130877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fcd00 0x7f6e000fd150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6df0004940 tx=0x7f6df0009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 -- 172.21.15.134:0/3229130877 shutdown_connections 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 --2- 172.21.15.134:0/3229130877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fcd00 0x7f6e000fd150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 --2- 172.21.15.134:0/3229130877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fbaf0 0x7f6e000fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 -- 172.21.15.134:0/3229130877 >> 172.21.15.134:0/3229130877 conn(0x7f6e000f7190 msgr2=0x7f6e000f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 -- 172.21.15.134:0/3229130877 shutdown_connections 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.194+0000 7f6e05351700 1 -- 172.21.15.134:0/3229130877 wait complete. 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 Processor -- start 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 -- start start 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fcd00 0x7f6e001127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00066110 con 0x7f6e000fbaf0 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6e05351700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00066250 con 0x7f6e000fcd00 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:52.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47112/0 (socket says 172.21.15.134:47112) 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 -- 172.21.15.134:0/412066213 learned_addr learned my addr 172.21.15.134:0/412066213 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfe7fc700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fcd00 0x7f6e001127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 -- 172.21.15.134:0/412066213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fcd00 msgr2=0x7f6e001127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fcd00 0x7f6e001127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 -- 172.21.15.134:0/412066213 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0005040 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.195+0000 7f6dfeffd700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6de8002a80 tx=0x7f6de8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.196+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de8007cc0 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.196+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6de800f040 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.196+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e00112de0 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.196+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de800d9c0 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.196+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e00113240 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.197+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0004bb80 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.197+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f6de800db40 con 0x7f6e000fbaf0 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.197+0000 7f6df77fe700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 0x7f6dec06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:52.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.197+0000 7f6dfe7fc700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 0x7f6dec06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:52.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.197+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6de8086af0 con 0x7f6e000fbaf0 2024-01-19T06:24:52.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.198+0000 7f6dfe7fc700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 0x7f6dec06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6df000b3b0 tx=0x7f6df0009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:52.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.202+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6de8052f40 con 0x7f6e000fbaf0 2024-01-19T06:24:52.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:52.434+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f6e001134c0 con 0x7f6e000fbaf0 2024-01-19T06:24:52.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:52 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.134:6826/3512441553,v1:172.21.15.134:6827/3512441553] up:active 2024-01-19T06:24:52.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:52 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 2024-01-19T06:24:52.674 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:52.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:52 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.134:6826/3512441553,v1:172.21.15.134:6827/3512441553] up:active 2024-01-19T06:24:52.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:52 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 2024-01-19T06:24:52.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:24:53.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.313+0000 7f6df77fe700 1 -- 172.21.15.134:0/412066213 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6de8056560 con 0x7f6e000fbaf0 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.315+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 msgr2=0x7f6dec06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.315+0000 7f6e05351700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 0x7f6dec06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6df000b3b0 tx=0x7f6df0009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.315+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 msgr2=0x7f6e00110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.315+0000 7f6e05351700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6de8002a80 tx=0x7f6de8004600 comp rx=0 tx=0).stop 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 shutdown_connections 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f6dec068f60 0x7f6dec06b410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e000fbaf0 0x7f6e00110290 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 --2- 172.21.15.134:0/412066213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e000fcd00 0x7f6e001127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 >> 172.21.15.134:0/412066213 conn(0x7f6e000f7190 msgr2=0x7f6e000fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 shutdown_connections 2024-01-19T06:24:53.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:53.316+0000 7f6e05351700 1 -- 172.21.15.134:0/412066213 wait complete. 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/412066213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.chvejw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.chvejw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:53.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:53.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: Deploying daemon mds.cephfs.smithi134.chvejw on smithi134 2024-01-19T06:24:53.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:53 smithi134 ceph-mon[28283]: pgmap v176: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 421 B/s wr, 1 op/s 2024-01-19T06:24:53.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/412066213' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.chvejw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.chvejw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: Deploying daemon mds.cephfs.smithi134.chvejw on smithi134 2024-01-19T06:24:53.683 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:53 smithi150 ceph-mon[35328]: pgmap v176: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 421 B/s wr, 1 op/s 2024-01-19T06:24:53.976 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:53.986 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:53.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-01-19T06:24:54.587 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:54.637 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:54 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:boot 2024-01-19T06:24:54.637 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:54 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/412066213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-01-19T06:24:54.637 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:54 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 1 up:standby 2024-01-19T06:24:54.637 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:54 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.xqfgpx"}]: dispatch 2024-01-19T06:24:54.637 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:54 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 1 up:standby 2024-01-19T06:24:54.644 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:54 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:boot 2024-01-19T06:24:54.644 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:54 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/412066213' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-01-19T06:24:54.644 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:54 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 1 up:standby 2024-01-19T06:24:54.644 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:54 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.xqfgpx"}]: dispatch 2024-01-19T06:24:54.644 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:54 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 1 up:standby 2024-01-19T06:24:55.626 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:55 smithi134 ceph-mon[28283]: pgmap v177: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 741 B/s wr, 2 op/s 2024-01-19T06:24:55.682 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:55 smithi150 ceph-mon[35328]: pgmap v177: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 741 B/s wr, 2 op/s 2024-01-19T06:24:56.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 -- 172.21.15.134:0/921797072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 msgr2=0x7f99a80fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:56.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 --2- 172.21.15.134:0/921797072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a80fe3b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f99a4004940 tx=0x7f99a4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:56.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 -- 172.21.15.134:0/921797072 shutdown_connections 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 --2- 172.21.15.134:0/921797072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a80fe3b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 --2- 172.21.15.134:0/921797072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 0x7f99a80fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 -- 172.21.15.134:0/921797072 >> 172.21.15.134:0/921797072 conn(0x7f99a80f6950 msgr2=0x7f99a80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 -- 172.21.15.134:0/921797072 shutdown_connections 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.449+0000 7f99b0083700 1 -- 172.21.15.134:0/921797072 wait complete. 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 Processor -- start 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 -- start start 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 0x7f99a810d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a810e190 con 0x7f99a80fdf60 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99b0083700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a810ea30 con 0x7f99a80faed0 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99ad61e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:56.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99ad61e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47122/0 (socket says 172.21.15.134:47122) 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99ad61e700 1 -- 172.21.15.134:0/3932886353 learned_addr learned my addr 172.21.15.134:0/3932886353 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.450+0000 7f99ade1f700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 0x7f99a810d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99ad61e700 1 -- 172.21.15.134:0/3932886353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 msgr2=0x7f99a810d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99ad61e700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 0x7f99a810d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99ad61e700 1 -- 172.21.15.134:0/3932886353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4005040 con 0x7f99a80fdf60 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99ad61e700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99a400b3b0 tx=0x7f99a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99a400e040 con 0x7f99a80fdf60 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a810ecb0 con 0x7f99a80fdf60 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99a4013420 con 0x7f99a80fdf60 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99a401c650 con 0x7f99a80fdf60 2024-01-19T06:24:56.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.451+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a810f1f0 con 0x7f99a80fdf60 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.452+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a81075f0 con 0x7f99a80fdf60 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.452+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f99a4013590 con 0x7f99a80fdf60 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.453+0000 7f999e7fc700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 0x7f999406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.453+0000 7f99ade1f700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 0x7f999406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.453+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f99a4088120 con 0x7f99a80fdf60 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.453+0000 7f99ade1f700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 0x7f999406b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99a8065960 tx=0x7f999800b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:24:56.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.457+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f99a4058130 con 0x7f99a80fdf60 2024-01-19T06:24:56.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:56.697+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f99a804bb80 con 0x7f99a80fdf60 2024-01-19T06:24:57.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.624+0000 7f999e7fc700 1 -- 172.21.15.134:0/3932886353 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f99a804bb80 con 0x7f99a80fdf60 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 msgr2=0x7f999406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 0x7f999406b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99a8065960 tx=0x7f999800b040 comp rx=0 tx=0).stop 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 msgr2=0x7f99a810db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f99a400b3b0 tx=0x7f99a4009e30 comp rx=0 tx=0).stop 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 shutdown_connections 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9994068f60 0x7f999406b410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80faed0 0x7f99a810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 --2- 172.21.15.134:0/3932886353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99a80fdf60 0x7f99a810db90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 >> 172.21.15.134:0/3932886353 conn(0x7f99a80f6950 msgr2=0x7f99a80f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 shutdown_connections 2024-01-19T06:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:24:57.627+0000 7f99b0083700 1 -- 172.21.15.134:0/3932886353 wait complete. 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3932886353' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.tsxxhg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.tsxxhg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:57.732 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:57.733 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:57 smithi150 ceph-mon[35328]: Deploying daemon mds.cephfs.smithi150.tsxxhg on smithi150 2024-01-19T06:24:57.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T06:24:57.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3932886353' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-01-19T06:24:57.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:57.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.tsxxhg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T06:24:57.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi150.tsxxhg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T06:24:57.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:24:57.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:57 smithi134 ceph-mon[28283]: Deploying daemon mds.cephfs.smithi150.tsxxhg on smithi150 2024-01-19T06:24:58.458 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:24:58.468 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:24:58.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-01-19T06:24:58.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:58 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3932886353' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-01-19T06:24:58.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:58 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.134:6828/3807665507,v1:172.21.15.134:6829/3807665507] up:boot 2024-01-19T06:24:58.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:58 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 2 up:standby 2024-01-19T06:24:58.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:58 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.chvejw"}]: dispatch 2024-01-19T06:24:58.738 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:24:58.809 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:58 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3932886353' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-01-19T06:24:58.809 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:58 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.134:6828/3807665507,v1:172.21.15.134:6829/3807665507] up:boot 2024-01-19T06:24:58.809 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:58 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 2 up:standby 2024-01-19T06:24:58.809 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:58 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.chvejw"}]: dispatch 2024-01-19T06:24:59.901 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:59 smithi134 ceph-mon[28283]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-01-19T06:24:59.901 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:59.901 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:24:59 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:59 smithi150 ceph-mon[35328]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-01-19T06:24:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:24:59.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:24:59 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 -- 172.21.15.134:0/2545306563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fde00 msgr2=0x7f7a200fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2545306563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fde00 0x7f7a200fe250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c004940 tx=0x7f7a1c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 -- 172.21.15.134:0/2545306563 shutdown_connections 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2545306563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fde00 0x7f7a200fe250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2545306563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fcbf0 0x7f7a200fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 -- 172.21.15.134:0/2545306563 >> 172.21.15.134:0/2545306563 conn(0x7f7a200f82d0 msgr2=0x7f7a200fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 -- 172.21.15.134:0/2545306563 shutdown_connections 2024-01-19T06:25:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.463+0000 7f7a27c59700 1 -- 172.21.15.134:0/2545306563 wait complete. 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 Processor -- start 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 -- start start 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fde00 0x7f7a2010d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2010de10 con 0x7f7a200fcbf0 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a27c59700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2010e6b0 con 0x7f7a200fde00 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.464+0000 7f7a259f5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49568/0 (socket says 172.21.15.134:49568) 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 -- 172.21.15.134:0/2286413860 learned_addr learned my addr 172.21.15.134:0/2286413860 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a251f4700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fde00 0x7f7a2010d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 -- 172.21.15.134:0/2286413860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fde00 msgr2=0x7f7a2010d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fde00 0x7f7a2010d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 -- 172.21.15.134:0/2286413860 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c005040 con 0x7f7a200fcbf0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a259f5700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7a10002a80 tx=0x7f7a100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a10007cc0 con 0x7f7a200fcbf0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a1000f040 con 0x7f7a200fcbf0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2010e990 con 0x7f7a200fcbf0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a1000d9e0 con 0x7f7a200fcbf0 2024-01-19T06:25:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.465+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2010eed0 con 0x7f7a200fcbf0 2024-01-19T06:25:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.466+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2004bb80 con 0x7f7a200fcbf0 2024-01-19T06:25:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.467+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f7a1000dc50 con 0x7f7a200fcbf0 2024-01-19T06:25:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.467+0000 7f7a167fc700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 0x7f7a0c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.468+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7a100865d0 con 0x7f7a200fcbf0 2024-01-19T06:25:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.468+0000 7f7a251f4700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 0x7f7a0c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.468+0000 7f7a251f4700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 0x7f7a0c06b410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00b3b0 tx=0x7f7a1c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.472+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7a1004ca00 con 0x7f7a200fcbf0 2024-01-19T06:25:00.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:00.725+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7a20105590 con 0x7f7a200fcbf0 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6826/2769186085,v1:172.21.15.150:6827/2769186085] up:boot 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 3 up:standby 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.tsxxhg"}]: dispatch 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:00 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:25:00.932 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6826/2769186085,v1:172.21.15.150:6827/2769186085] up:boot 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 3 up:standby 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi150.tsxxhg"}]: dispatch 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:00.933 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:00 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:25:01.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.690+0000 7f7a167fc700 1 -- 172.21.15.134:0/2286413860 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7a100564b0 con 0x7f7a200fcbf0 2024-01-19T06:25:01.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 msgr2=0x7f7a0c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:01.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 0x7f7a0c06b410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00b3b0 tx=0x7f7a1c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:01.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 msgr2=0x7f7a2010d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7a10002a80 tx=0x7f7a100046b0 comp rx=0 tx=0).stop 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 shutdown_connections 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7a0c068f60 0x7f7a0c06b410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a200fcbf0 0x7f7a2010d2d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 --2- 172.21.15.134:0/2286413860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a200fde00 0x7f7a2010d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 >> 172.21.15.134:0/2286413860 conn(0x7f7a200f82d0 msgr2=0x7f7a20101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 shutdown_connections 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:01.692+0000 7f7a27c59700 1 -- 172.21.15.134:0/2286413860 wait complete. 2024-01-19T06:25:01.693 INFO:teuthology.orchestra.run.smithi134.stderr:inline data enabled 2024-01-19T06:25:01.909 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-19T06:25:01.909 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:standby 2024-01-19T06:25:01.909 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: Dropping low affinity active daemon mds.cephfs.smithi134.higalz in favor of higher affinity standby. 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: Replacing daemon mds.cephfs.smithi134.higalz as rank 0 with standby daemon mds.cephfs.smithi150.xqfgpx 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 3 up:standby 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: osdmap e46: 6 total, 6 up, 6 in 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:replay} 2 up:standby 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2286413860' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-19T06:25:01.910 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:01 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:02.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:standby 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: Dropping low affinity active daemon mds.cephfs.smithi134.higalz in favor of higher affinity standby. 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: Replacing daemon mds.cephfs.smithi134.higalz as rank 0 with standby daemon mds.cephfs.smithi150.xqfgpx 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi134.higalz=up:active} 3 up:standby 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: osdmap e46: 6 total, 6 up, 6 in 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:replay} 2 up:standby 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2286413860' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-19T06:25:02.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:01 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:02.778 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T06:25:02.788 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:25:02.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-19T06:25:03.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:reconnect 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2286413860' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.134:6826/1697349819,v1:172.21.15.134:6827/1697349819] up:boot 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.134:6828/3807665507,v1:172.21.15.134:6829/3807665507] up:standby 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:reconnect} 3 up:standby 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.higalz"}]: dispatch 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:02 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:03.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:reconnect 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2286413860' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.134:6826/1697349819,v1:172.21.15.134:6827/1697349819] up:boot 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.134:6828/3807665507,v1:172.21.15.134:6829/3807665507] up:standby 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:reconnect} 3 up:standby 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.higalz"}]: dispatch 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:03.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:02 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:03.486 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:03 smithi134 ceph-mon[28283]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 1023 B/s wr, 3 op/s 2024-01-19T06:25:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:03 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:rejoin 2024-01-19T06:25:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:03 smithi134 ceph-mon[28283]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:rejoin} 3 up:standby 2024-01-19T06:25:03.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:03 smithi134 ceph-mon[28283]: daemon mds.cephfs.smithi150.xqfgpx is now active in filesystem cephfs as rank 0 2024-01-19T06:25:04.139 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:03 smithi150 ceph-mon[35328]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s rd, 1023 B/s wr, 3 op/s 2024-01-19T06:25:04.140 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:03 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:rejoin 2024-01-19T06:25:04.140 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:03 smithi150 ceph-mon[35328]: fsmap cephfs:1/1 {0=cephfs.smithi150.xqfgpx=up:rejoin} 3 up:standby 2024-01-19T06:25:04.140 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:03 smithi150 ceph-mon[35328]: daemon mds.cephfs.smithi150.xqfgpx is now active in filesystem cephfs as rank 0 2024-01-19T06:25:04.874 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:04 smithi134 ceph-mon[28283]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-19T06:25:04.928 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:04 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:active 2024-01-19T06:25:04.928 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:04 smithi134 ceph-mon[28283]: mds.? [v2:172.21.15.150:6826/2769186085,v1:172.21.15.150:6827/2769186085] up:standby 2024-01-19T06:25:04.928 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:04 smithi134 ceph-mon[28283]: fsmap cephfs:1 {0=cephfs.smithi150.xqfgpx=up:active} 3 up:standby 2024-01-19T06:25:05.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:04 smithi150 ceph-mon[35328]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-19T06:25:05.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:04 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] up:active 2024-01-19T06:25:05.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:04 smithi150 ceph-mon[35328]: mds.? [v2:172.21.15.150:6826/2769186085,v1:172.21.15.150:6827/2769186085] up:standby 2024-01-19T06:25:05.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:04 smithi150 ceph-mon[35328]: fsmap cephfs:1 {0=cephfs.smithi150.xqfgpx=up:active} 3 up:standby 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 -- 172.21.15.134:0/2699162943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fcd00 msgr2=0x7fac500fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 --2- 172.21.15.134:0/2699162943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fcd00 0x7fac500fd150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac40004940 tx=0x7fac40009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 -- 172.21.15.134:0/2699162943 shutdown_connections 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 --2- 172.21.15.134:0/2699162943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fcd00 0x7fac500fd150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 --2- 172.21.15.134:0/2699162943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fbaf0 0x7fac500fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:05.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.894+0000 7fac58028700 1 -- 172.21.15.134:0/2699162943 >> 172.21.15.134:0/2699162943 conn(0x7fac500f71d0 msgr2=0x7fac500f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 -- 172.21.15.134:0/2699162943 shutdown_connections 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 -- 172.21.15.134:0/2699162943 wait complete. 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 Processor -- start 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 -- start start 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fcd00 0x7fac5010c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5010cf50 con 0x7fac500fbaf0 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.895+0000 7fac58028700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5010d7f0 con 0x7fac500fcd00 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49596/0 (socket says 172.21.15.134:49596) 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 -- 172.21.15.134:0/1912872270 learned_addr learned my addr 172.21.15.134:0/1912872270 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac555c3700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fcd00 0x7fac5010c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:05.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 -- 172.21.15.134:0/1912872270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fcd00 msgr2=0x7fac5010c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fcd00 0x7fac5010c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 -- 172.21.15.134:0/1912872270 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac40005040 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac55dc4700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fac4c002a50 tx=0x7fac4c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4c007cc0 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac4c00f040 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5010da70 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4c00d8d0 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.896+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5010dfb0 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.897+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5004bb80 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.897+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fac4c00dab0 con 0x7fac500fbaf0 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.898+0000 7fac467fc700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 0x7fac3c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:06.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.898+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fac4c086aa0 con 0x7fac500fbaf0 2024-01-19T06:25:06.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.898+0000 7fac555c3700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 0x7fac3c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:06.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.898+0000 7fac555c3700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 0x7fac3c06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b3b0 tx=0x7fac40009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:06.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:05.902+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fac4c0569b0 con 0x7fac500fbaf0 2024-01-19T06:25:06.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:05 smithi134 ceph-mon[28283]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 716 B/s wr, 3 op/s 2024-01-19T06:25:06.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.132+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fac5010e430 con 0x7fac500fbaf0 2024-01-19T06:25:06.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.133+0000 7fac467fc700 1 -- 172.21.15.134:0/1912872270 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1813 (secure 0 0 0) 0x7fac4c056540 con 0x7fac500fbaf0 2024-01-19T06:25:06.133 INFO:teuthology.orchestra.run.smithi134.stdout:e14 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:legacy client fscid: 1 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:Filesystem 'cephfs' (1) 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:fs_name cephfs 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 14 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-01-19T06:24:45.276806+0000 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:modified 2024-01-19T06:25:03.694773+0000 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:tableserver 0 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:root 0 2024-01-19T06:25:06.134 INFO:teuthology.orchestra.run.smithi134.stdout:session_timeout 60 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:session_autoclose 300 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:max_file_size 1099511627776 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:required_client_features {} 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure 0 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure_osd_epoch 46 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:max_mds 1 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:in 0 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:up {0=24255} 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:failed 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:damaged 2024-01-19T06:25:06.135 INFO:teuthology.orchestra.run.smithi134.stdout:stopped 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:data_pools [3] 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:metadata_pool 2 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:inline_data enabled 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:balancer 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:standby_count_wanted 1 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.cephfs.smithi150.xqfgpx{0:24255} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.150:6824/379459445,v1:172.21.15.150:6825/379459445] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:Standby daemons: 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.cephfs.smithi134.chvejw{-1:14474} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.134:6828/3807665507,v1:172.21.15.134:6829/3807665507] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T06:25:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.cephfs.smithi134.higalz{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1697349819,v1:172.21.15.134:6827/1697349819] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T06:25:06.137 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.cephfs.smithi150.tsxxhg{-1:24267} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.150:6826/2769186085,v1:172.21.15.150:6827/2769186085] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T06:25:06.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 msgr2=0x7fac3c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:06.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 0x7fac3c06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b3b0 tx=0x7fac40009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 msgr2=0x7fac5010c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fac4c002a50 tx=0x7fac4c004600 comp rx=0 tx=0).stop 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 shutdown_connections 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fac3c068f60 0x7fac3c06b410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac500fbaf0 0x7fac5010c410 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 --2- 172.21.15.134:0/1912872270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac500fcd00 0x7fac5010c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 >> 172.21.15.134:0/1912872270 conn(0x7fac500f71d0 msgr2=0x7fac500fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 shutdown_connections 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:06.135+0000 7fac58028700 1 -- 172.21.15.134:0/1912872270 wait complete. 2024-01-19T06:25:06.138 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 14 2024-01-19T06:25:06.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:05 smithi150 ceph-mon[35328]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 716 B/s wr, 3 op/s 2024-01-19T06:25:06.887 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:06 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1912872270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T06:25:06.887 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:06.887 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:06 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:06.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-01-19T06:25:07.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:06 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1912872270' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T06:25:07.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:07.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:06 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:08.310 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:08.310 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.310 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.311 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:25:08.311 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:25:08.311 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.311 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:08 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:25:08.317 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:08.317 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.317 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.317 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:25:08.317 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:25:08.318 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:08.318 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:08 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:25:08.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 -- 172.21.15.134:0/3679898449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 msgr2=0x7f7c0406ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:08.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 --2- 172.21.15.134:0/3679898449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0406ffd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8004940 tx=0x7f7bf8009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:08.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 -- 172.21.15.134:0/3679898449 shutdown_connections 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 --2- 172.21.15.134:0/3679898449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c04070980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 --2- 172.21.15.134:0/3679898449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0406ffd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 -- 172.21.15.134:0/3679898449 >> 172.21.15.134:0/3679898449 conn(0x7f7c040f82d0 msgr2=0x7f7c040fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 -- 172.21.15.134:0/3679898449 shutdown_connections 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.846+0000 7f7c093eb700 1 -- 172.21.15.134:0/3679898449 wait complete. 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.847+0000 7f7c093eb700 1 Processor -- start 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.847+0000 7f7c093eb700 1 -- start start 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.847+0000 7f7c093eb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.847+0000 7f7c093eb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c093eb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041102d0 con 0x7f7c04071b60 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c093eb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04110b70 con 0x7f7c04070510 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c02ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c02ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:49698/0 (socket says 172.21.15.134:49698) 2024-01-19T06:25:08.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c02ffd700 1 -- 172.21.15.134:0/2532370189 learned_addr learned my addr 172.21.15.134:0/2532370189 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44942/0 (socket says 172.21.15.134:44942) 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 -- 172.21.15.134:0/2532370189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 msgr2=0x7f7c0410f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 -- 172.21.15.134:0/2532370189 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8005040 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c02ffd700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c027fc700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4002820 tx=0x7f7bf4004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf4007cc0 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bf400f040 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c04110e50 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bf40137d0 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.848+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c04111390 con 0x7f7c04071b60 2024-01-19T06:25:08.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.849+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c041096f0 con 0x7f7c04071b60 2024-01-19T06:25:08.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.850+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f7bf401c400 con 0x7f7c04071b60 2024-01-19T06:25:08.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.850+0000 7f7beb7fe700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 0x7f7bec06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:08.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.850+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7bf4086ac0 con 0x7f7c04071b60 2024-01-19T06:25:08.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.850+0000 7f7c02ffd700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 0x7f7bec06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:08.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.850+0000 7f7c02ffd700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 0x7f7bec06b420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800b3b0 tx=0x7f7bf8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:08.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:08.854+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7bf4056950 con 0x7f7c04071b60 2024-01-19T06:25:09.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.099+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0404bb80 con 0x7f7c04071b60 2024-01-19T06:25:09.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.100+0000 7f7beb7fe700 1 -- 172.21.15.134:0/2532370189 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4697 (secure 0 0 0) 0x7f7c0404bb80 con 0x7f7c04071b60 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 msgr2=0x7f7bec06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 0x7f7bec06b420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800b3b0 tx=0x7f7bf8009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 msgr2=0x7f7c0410fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4002820 tx=0x7f7bf4004570 comp rx=0 tx=0).stop 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 shutdown_connections 2024-01-19T06:25:09.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7bec068f70 0x7f7bec06b420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c04070510 0x7f7c0410f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 --2- 172.21.15.134:0/2532370189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c04071b60 0x7f7c0410fcd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 >> 172.21.15.134:0/2532370189 conn(0x7f7c040f82d0 msgr2=0x7f7c04103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 shutdown_connections 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:09.102+0000 7f7c093eb700 1 -- 172.21.15.134:0/2532370189 wait complete. 2024-01-19T06:25:09.103 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 14 2024-01-19T06:25:09.112 INFO:teuthology.orchestra.run.smithi134.stdout:true 2024-01-19T06:25:10.280 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:10 smithi134 ceph-mon[28283]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:10.281 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:10 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2532370189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T06:25:10.281 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:10.281 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:10 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:10.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-01-19T06:25:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:10 smithi150 ceph-mon[35328]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:10 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2532370189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T06:25:10.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:10.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:10 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:11.043 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:12.418 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:12 smithi134 ceph-mon[28283]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:12.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:12 smithi150 ceph-mon[35328]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:13.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 -- 172.21.15.134:0/2703223566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd18066090 msgr2=0x7efd180ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:13.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 --2- 172.21.15.134:0/2703223566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd18066090 0x7efd180ff7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004940 tx=0x7efd0c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:13.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 -- 172.21.15.134:0/2703223566 shutdown_connections 2024-01-19T06:25:13.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 --2- 172.21.15.134:0/2703223566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd180ffd70 0x7efd181021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:13.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 --2- 172.21.15.134:0/2703223566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd18066090 0x7efd180ff7a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 -- 172.21.15.134:0/2703223566 >> 172.21.15.134:0/2703223566 conn(0x7efd180f7190 msgr2=0x7efd180f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 -- 172.21.15.134:0/2703223566 shutdown_connections 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 -- 172.21.15.134:0/2703223566 wait complete. 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.607+0000 7efd1dc05700 1 Processor -- start 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd1dc05700 1 -- start start 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd1dc05700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd18066090 0x7efd1810d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd1dc05700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd1dc05700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1810e0c0 con 0x7efd180ffd70 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd1dc05700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1810e960 con 0x7efd18066090 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44952/0 (socket says 172.21.15.134:44952) 2024-01-19T06:25:13.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 -- 172.21.15.134:0/88961717 learned_addr learned my addr 172.21.15.134:0/88961717 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:13.929 INFO:teuthology.orchestra.run.smithi134.stdout:true 2024-01-19T06:25:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd177fe700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd18066090 0x7efd1810d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 -- 172.21.15.134:0/88961717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd18066090 msgr2=0x7efd1810d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd18066090 0x7efd1810d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:13.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 -- 172.21.15.134:0/88961717 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c005040 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.608+0000 7efd16ffd700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c800 tx=0x7efd08004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.609+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd080106f0 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.609+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1810ec40 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.609+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd08010d00 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.609+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd080113a0 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.609+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1810f180 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.610+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd181075e0 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.610+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7efd08012400 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.610+0000 7efcfffff700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 0x7efd0006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.610+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7efd08086d70 con 0x7efd180ffd70 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.610+0000 7efd177fe700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 0x7efd0006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:13.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.611+0000 7efd177fe700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 0x7efd0006b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004890 tx=0x7efd0c009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.615+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7efd08056c80 con 0x7efd180ffd70 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.886+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7efd1804bb80 con 0x7efd180ffd70 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.887+0000 7efcfffff700 1 -- 172.21.15.134:0/88961717 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v14) v1 ==== 78+0+99 (secure 0 0 0) 0x7efd08056810 con 0x7efd180ffd70 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 msgr2=0x7efd0006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 0x7efd0006b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004890 tx=0x7efd0c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 msgr2=0x7efd1810dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c800 tx=0x7efd08004470 comp rx=0 tx=0).stop 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 shutdown_connections 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7efd00068f60 0x7efd0006b410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd18066090 0x7efd1810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:13.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 --2- 172.21.15.134:0/88961717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd180ffd70 0x7efd1810dac0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd0800c800 tx=0x7efd08004470 comp rx=0 tx=0).stop 2024-01-19T06:25:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 >> 172.21.15.134:0/88961717 conn(0x7efd180f7190 msgr2=0x7efd180f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 shutdown_connections 2024-01-19T06:25:13.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:13.889+0000 7efd1dc05700 1 -- 172.21.15.134:0/88961717 wait complete. 2024-01-19T06:25:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:14 smithi134 ceph-mon[28283]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:14.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:14 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/88961717' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-19T06:25:14.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:14 smithi150 ceph-mon[35328]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T06:25:14.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:14 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/88961717' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-19T06:25:14.616 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-01-19T06:25:14.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T06:25:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 -- 172.21.15.134:0/2487623456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44800ff0e0 msgr2=0x7f44801014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 --2- 172.21.15.134:0/2487623456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44800ff0e0 0x7f44801014c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f44700048d0 tx=0x7f4470009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 -- 172.21.15.134:0/2487623456 shutdown_connections 2024-01-19T06:25:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 --2- 172.21.15.134:0/2487623456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44800ff0e0 0x7f44801014c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 --2- 172.21.15.134:0/2487623456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4480066000 0x7f44800feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 -- 172.21.15.134:0/2487623456 >> 172.21.15.134:0/2487623456 conn(0x7f44800f65f0 msgr2=0x7f44800f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 -- 172.21.15.134:0/2487623456 shutdown_connections 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.741+0000 7f4487407640 1 -- 172.21.15.134:0/2487623456 wait complete. 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 Processor -- start 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 -- start start 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44800ff0e0 0x7f448010cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448010d2e0 con 0x7f4480066000 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f4487407640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480111d30 con 0x7f44800ff0e0 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44974/0 (socket says 172.21.15.134:44974) 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 -- 172.21.15.134:0/1511798635 learned_addr learned my addr 172.21.15.134:0/1511798635 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44849a2640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44800ff0e0 0x7f448010cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 -- 172.21.15.134:0/1511798635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44800ff0e0 msgr2=0x7f448010cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44800ff0e0 0x7f448010cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 -- 172.21.15.134:0/1511798635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470005040 con 0x7f4480066000 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.742+0000 7f44851a3640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c7e0 tx=0x7f447c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.743+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447c00a820 con 0x7f4480066000 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.743+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f447c00ae30 con 0x7f4480066000 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.743+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4480111ed0 con 0x7f4480066000 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.743+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447c0113a0 con 0x7f4480066000 2024-01-19T06:25:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.743+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44801123e0 con 0x7f4480066000 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.744+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480104970 con 0x7f4480066000 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.744+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f447c00a980 con 0x7f4480066000 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.744+0000 7f4475ffb640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 0x7f446c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.744+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f447c086fa0 con 0x7f4480066000 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.744+0000 7f44849a2640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 0x7f446c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.745+0000 7f44849a2640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 0x7f446c06b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4470009a10 tx=0x7f4470009da0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.749+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f447c0534e0 con 0x7f4480066000 2024-01-19T06:25:14.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.919+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4480112bf0 con 0x7f4480066000 2024-01-19T06:25:14.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.921+0000 7f4475ffb640 1 -- 172.21.15.134:0/1511798635 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14440 (secure 0 0 0) 0x7f447c056b00 con 0x7f4480066000 2024-01-19T06:25:14.921 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:14.921 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":46,"fsid":"573cd99a-b692-11ee-95ae-87774f69a715","created":"2024-01-19T06:18:02.238207+0000","modified":"2024-01-19T06:25:00.685614+0000","last_up_change":"2024-01-19T06:23:27.392310+0000","last_in_change":"2024-01-19T06:23:08.932614+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T06:22:14.846038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-01-19T06:24:44.215142+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-01-19T06:24:44.507175+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6866d2e8-d629-4fd8-ab59-37109281459b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6803","nonce":3497975570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6805","nonce":3497975570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6809","nonce":3497975570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3497975570},{"type":"v1","addr":"172.21.15.134:6807","nonce":3497975570}]},"public_addr":"172.21.15.134:6803/3497975570","cluster_addr":"172.21.15.134:6805/3497975570","heartbeat_back_addr":"172.21.15.134:6809/3497975570","heartbeat_front_addr":"172.21.15.134:6807/3497975570","state":["exists","up"]},{"osd":1,"uuid":"c8d35ff7-48c7-49f5-afa6-e43f5d10212f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6811","nonce":743565436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6813","nonce":743565436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6817","nonce":743565436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":743565436},{"type":"v1","addr":"172.21.15.134:6815","nonce":743565436}]},"public_addr":"172.21.15.134:6811/743565436","cluster_addr":"172.21.15.134:6813/743565436","heartbeat_back_addr":"172.21.15.134:6817/743565436","heartbeat_front_addr":"172.21.15.134:6815/743565436","state":["exists","up"]},{"osd":2,"uuid":"b2df8d4e-2a3c-479a-8033-2b777cf50888","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6819","nonce":3103353719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6821","nonce":3103353719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6825","nonce":3103353719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3103353719},{"type":"v1","addr":"172.21.15.134:6823","nonce":3103353719}]},"public_addr":"172.21.15.134:6819/3103353719","cluster_addr":"172.21.15.134:6821/3103353719","heartbeat_back_addr":"172.21.15.134:6825/3103353719","heartbeat_front_addr":"172.21.15.134:6823/3103353719","state":["exists","up"]},{"osd":3,"uuid":"b66525f9-4493-4708-bbf7-e76de91fd5b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6801","nonce":4178102778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6803","nonce":4178102778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6807","nonce":4178102778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":4178102778},{"type":"v1","addr":"172.21.15.150:6805","nonce":4178102778}]},"public_addr":"172.21.15.150:6801/4178102778","cluster_addr":"172.21.15.150:6803/4178102778","heartbeat_back_addr":"172.21.15.150:6807/4178102778","heartbeat_front_addr":"172.21.15.150:6805/4178102778","state":["exists","up"]},{"osd":4,"uuid":"6efffbf8-04e9-4b01-8723-a6837b2bdb63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6809","nonce":996513041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6811","nonce":996513041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6815","nonce":996513041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":996513041},{"type":"v1","addr":"172.21.15.150:6813","nonce":996513041}]},"public_addr":"172.21.15.150:6809/996513041","cluster_addr":"172.21.15.150:6811/996513041","heartbeat_back_addr":"172.21.15.150:6815/996513041","heartbeat_front_addr":"172.21.15.150:6813/996513041","state":["exists","up"]},{"osd":5,"uuid":"8e42646f-77fc-418a-9339-79b1130d4906","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6817","nonce":2546848737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6819","nonce":2546848737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6823","nonce":2546848737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2546848737},{"type":"v1","addr":"172.21.15.150:6821","nonce":2546848737}]},"public_addr":"172.21.15.150:6817/2546848737","cluster_addr":"172.21.15.150:6819/2546848737","heartbeat_back_addr":"172.21.15.150:6823/2546848737","heartbeat_front_addr":"172.21.15.150:6821/2546848737","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:16.885118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:21:45.420119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:12.603236+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:22:35.079173+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:00.172610+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T06:23:26.088382+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:6827/3512441553":"2024-01-20T06:25:00.685587+0000","172.21.15.134:0/620851043":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1593458977":"2024-01-20T06:20:06.516885+0000","172.21.15.134:0/1277951871":"2024-01-20T06:18:33.202902+0000","172.21.15.134:6800/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/3378221656":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/2486479390":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6826/3512441553":"2024-01-20T06:25:00.685587+0000","172.21.15.134:0/1811685245":"2024-01-20T06:18:33.202902+0000","172.21.15.134:6801/2771969091":"2024-01-20T06:18:33.202902+0000","172.21.15.134:0/1113248930":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6801/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:6800/4139544225":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/2127635670":"2024-01-20T06:19:03.684226+0000","172.21.15.134:0/3213922814":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6800/1934811542":"2024-01-20T06:20:06.516885+0000","172.21.15.134:6801/1934811542":"2024-01-20T06:20:06.516885+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T06:25:14.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 msgr2=0x7f446c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:14.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 0x7f446c06b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4470009a10 tx=0x7f4470009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 msgr2=0x7f448010c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:14.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c7e0 tx=0x7f447c0043c0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 shutdown_connections 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f446c068ed0 0x7f446c06b380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44800ff0e0 0x7f448010cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 --2- 172.21.15.134:0/1511798635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480066000 0x7f448010c750 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 >> 172.21.15.134:0/1511798635 conn(0x7f44800f65f0 msgr2=0x7f44800f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 shutdown_connections 2024-01-19T06:25:14.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:14.923+0000 7f4487407640 1 -- 172.21.15.134:0/1511798635 wait complete. 2024-01-19T06:25:14.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-19T06:22:14.846038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-01-19T06:24:44.215142+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-01-19T06:24:44.507175+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-19T06:25:14.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-19T06:25:15.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 -- 172.21.15.134:0/341582719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 msgr2=0x7f1e5c0fd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 --2- 172.21.15.134:0/341582719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c0fd740 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1e500048d0 tx=0x7f1e50009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 -- 172.21.15.134:0/341582719 shutdown_connections 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 --2- 172.21.15.134:0/341582719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c0fd740 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 --2- 172.21.15.134:0/341582719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 0x7f1e5c0fa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 -- 172.21.15.134:0/341582719 >> 172.21.15.134:0/341582719 conn(0x7f1e5c0f5d00 msgr2=0x7f1e5c0f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 -- 172.21.15.134:0/341582719 shutdown_connections 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.038+0000 7f1e6182e640 1 -- 172.21.15.134:0/341582719 wait complete. 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 Processor -- start 2024-01-19T06:25:15.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 -- start start 2024-01-19T06:25:15.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 0x7f1e5c10c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c10d2d0 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e6182e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c111d20 con 0x7f1e5c0fa220 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44992/0 (socket says 172.21.15.134:44992) 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 -- 172.21.15.134:0/2007982797 learned_addr learned my addr 172.21.15.134:0/2007982797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5affd640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 0x7f1e5c10c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 -- 172.21.15.134:0/2007982797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 msgr2=0x7f1e5c10c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 0x7f1e5c10c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 -- 172.21.15.134:0/2007982797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e50005040 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.039+0000 7f1e5a7fc640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002bda0 tx=0x7f1e50009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.040+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e500094c0 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.040+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e50009620 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.040+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c111e60 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.040+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e5003b400 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.040+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c1123d0 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.041+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c104980 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.041+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f1e5002f070 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.041+0000 7f1e437fe640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 0x7f1e4406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.041+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1e500a9470 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.041+0000 7f1e5affd640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 0x7f1e4406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.042+0000 7f1e5affd640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 0x7f1e4406b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009fd0 tx=0x7f1e4c009e50 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.046+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1e50075870 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:15 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1511798635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:25:15.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.216+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e5c1127e0 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.216+0000 7f1e437fe640 1 -- 172.21.15.134:0/2007982797 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e50078e90 con 0x7f1e5c0fd2d0 2024-01-19T06:25:15.217 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2024-01-19T06:25:15.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 msgr2=0x7f1e4406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 0x7f1e4406b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009fd0 tx=0x7f1e4c009e50 comp rx=0 tx=0).stop 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 msgr2=0x7f1e5c10cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002bda0 tx=0x7f1e50009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 shutdown_connections 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1e44068ed0 0x7f1e4406b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e5c0fd2d0 0x7f1e5c10cc80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 --2- 172.21.15.134:0/2007982797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e5c0fa220 0x7f1e5c10c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 >> 172.21.15.134:0/2007982797 conn(0x7f1e5c0f5d00 msgr2=0x7f1e5c100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 shutdown_connections 2024-01-19T06:25:15.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.218+0000 7f1e6182e640 1 -- 172.21.15.134:0/2007982797 wait complete. 2024-01-19T06:25:15.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/113137158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 msgr2=0x7f1ae006f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/113137158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae006f4e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1adc009c50 tx=0x7f1adc004d10 comp rx=0 tx=0).stop 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/113137158 shutdown_connections 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/113137158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 0x7f1ae006fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/113137158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae006f4e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/113137158 >> 172.21.15.134:0/113137158 conn(0x7f1ae00f76d0 msgr2=0x7f1ae00f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/113137158 shutdown_connections 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.329+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/113137158 wait complete. 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 Processor -- start 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 -- start start 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 0x7f1ae010e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae010f500 con 0x7f1ae0071050 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae0113f50 con 0x7f1ae006fa20 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45002/0 (socket says 172.21.15.134:45002) 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 -- 172.21.15.134:0/1932630483 learned_addr learned my addr 172.21.15.134:0/1932630483 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5d39640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 0x7f1ae010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 -- 172.21.15.134:0/1932630483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 msgr2=0x7f1ae010e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 0x7f1ae010e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 -- 172.21.15.134:0/1932630483 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc0098b0 con 0x7f1ae0071050 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae5538640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0002a50 tx=0x7f1ad0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad0007c30 con 0x7f1ae0071050 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ad000f040 con 0x7f1ae0071050 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ad0009430 con 0x7f1ae0071050 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae01140f0 con 0x7f1ae0071050 2024-01-19T06:25:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.330+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae0114660 con 0x7f1ae0071050 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.331+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0106ba0 con 0x7f1ae0071050 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.332+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f1ad00095f0 con 0x7f1ae0071050 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.332+0000 7f1ad67fc640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 0x7f1acc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.332+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1ad0086510 con 0x7f1ae0071050 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.332+0000 7f1ae5d39640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 0x7f1acc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.333+0000 7f1ae5d39640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 0x7f1acc06b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1adc003400 tx=0x7f1adc037040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.337+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1ad0052990 con 0x7f1ae0071050 2024-01-19T06:25:15.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:15 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1511798635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T06:25:15.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.549+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f1ae01148d0 con 0x7f1ae0071050 2024-01-19T06:25:15.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.550+0000 7f1ad67fc640 1 -- 172.21.15.134:0/1932630483 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1ad0055fb0 con 0x7f1ae0071050 2024-01-19T06:25:15.550 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 32 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 msgr2=0x7f1acc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 0x7f1acc06b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1adc003400 tx=0x7f1adc037040 comp rx=0 tx=0).stop 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 msgr2=0x7f1ae010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0002a50 tx=0x7f1ad0004600 comp rx=0 tx=0).stop 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 shutdown_connections 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1acc068ed0 0x7f1acc06b380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ae0071050 0x7f1ae010eeb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 --2- 172.21.15.134:0/1932630483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ae006fa20 0x7f1ae010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 >> 172.21.15.134:0/1932630483 conn(0x7f1ae00f76d0 msgr2=0x7f1ae0101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 shutdown_connections 2024-01-19T06:25:15.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.552+0000 7f1ae7f9d640 1 -- 172.21.15.134:0/1932630483 wait complete. 2024-01-19T06:25:15.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 -- 172.21.15.134:0/3245034821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fae40 msgr2=0x7f29ec0fb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 --2- 172.21.15.134:0/3245034821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fae40 0x7f29ec0fb250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f29e00048d0 tx=0x7f29e0009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 -- 172.21.15.134:0/3245034821 shutdown_connections 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 --2- 172.21.15.134:0/3245034821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fc060 0x7f29ec0fc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 --2- 172.21.15.134:0/3245034821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fae40 0x7f29ec0fb250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 -- 172.21.15.134:0/3245034821 >> 172.21.15.134:0/3245034821 conn(0x7f29ec0f65b0 msgr2=0x7f29ec0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 -- 172.21.15.134:0/3245034821 shutdown_connections 2024-01-19T06:25:15.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.670+0000 7f29f179d640 1 -- 172.21.15.134:0/3245034821 wait complete. 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 Processor -- start 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 -- start start 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fae40 0x7f29ec10b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec10c220 con 0x7f29ec0fc060 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29f179d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec110c70 con 0x7f29ec0fae40 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45012/0 (socket says 172.21.15.134:45012) 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 -- 172.21.15.134:0/4127839477 learned_addr learned my addr 172.21.15.134:0/4127839477 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29eaffd640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fae40 0x7f29ec10b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 -- 172.21.15.134:0/4127839477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fae40 msgr2=0x7f29ec10b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fae40 0x7f29ec10b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 -- 172.21.15.134:0/4127839477 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0005040 con 0x7f29ec0fc060 2024-01-19T06:25:15.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29ea7fc640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00eb70 tx=0x7f29dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.671+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29dc007c30 con 0x7f29ec0fc060 2024-01-19T06:25:15.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.672+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec110db0 con 0x7f29ec0fc060 2024-01-19T06:25:15.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.672+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec111320 con 0x7f29ec0fc060 2024-01-19T06:25:15.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.672+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29dc00f040 con 0x7f29ec0fc060 2024-01-19T06:25:15.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.672+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29dc009400 con 0x7f29ec0fc060 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.673+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d8005240 con 0x7f29ec0fc060 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.673+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f29dc009640 con 0x7f29ec0fc060 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.673+0000 7f29d37fe640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 0x7f29d406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.674+0000 7f29eaffd640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 0x7f29d406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.674+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f29dc086a10 con 0x7f29ec0fc060 2024-01-19T06:25:15.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.674+0000 7f29eaffd640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 0x7f29d406b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000c00 tx=0x7f29e0009da0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.678+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f29dc04da00 con 0x7f29ec0fc060 2024-01-19T06:25:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.842+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f29d8005a70 con 0x7f29ec0fc060 2024-01-19T06:25:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.842+0000 7f29d37fe640 1 -- 172.21.15.134:0/4127839477 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f29dc0190b0 con 0x7f29ec0fc060 2024-01-19T06:25:15.842 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 32 2024-01-19T06:25:15.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 msgr2=0x7f29d406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 0x7f29d406b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000c00 tx=0x7f29e0009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 msgr2=0x7f29ec10bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00eb70 tx=0x7f29dc00b040 comp rx=0 tx=0).stop 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 shutdown_connections 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f29d4068ed0 0x7f29d406b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29ec0fc060 0x7f29ec10bbd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 --2- 172.21.15.134:0/4127839477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29ec0fae40 0x7f29ec10b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 >> 172.21.15.134:0/4127839477 conn(0x7f29ec0f65b0 msgr2=0x7f29ec0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 shutdown_connections 2024-01-19T06:25:15.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.844+0000 7f29f179d640 1 -- 172.21.15.134:0/4127839477 wait complete. 2024-01-19T06:25:15.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 -- 172.21.15.134:0/1125282398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fae70 msgr2=0x7f766c0fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 --2- 172.21.15.134:0/1125282398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fae70 0x7f766c0fb280 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f765c0048d0 tx=0x7f765c009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 -- 172.21.15.134:0/1125282398 shutdown_connections 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 --2- 172.21.15.134:0/1125282398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fc090 0x7f766c0fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 --2- 172.21.15.134:0/1125282398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fae70 0x7f766c0fb280 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 -- 172.21.15.134:0/1125282398 >> 172.21.15.134:0/1125282398 conn(0x7f766c0f65c0 msgr2=0x7f766c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 -- 172.21.15.134:0/1125282398 shutdown_connections 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.952+0000 7f767199d640 1 -- 172.21.15.134:0/1125282398 wait complete. 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 Processor -- start 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 -- start start 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 0x7f766c065e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c111b00 con 0x7f766c0fc090 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c111c40 con 0x7f766c0fae70 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45030/0 (socket says 172.21.15.134:45030) 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f766affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 0x7f766c065e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 -- 172.21.15.134:0/789324119 learned_addr learned my addr 172.21.15.134:0/789324119 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 -- 172.21.15.134:0/789324119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 msgr2=0x7f766c065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 0x7f766c065e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 -- 172.21.15.134:0/789324119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c005040 con 0x7f766c0fc090 2024-01-19T06:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f766affd640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 0x7f766c065e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f7663fff640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f765400eb70 tx=0x7f765400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7654007c30 con 0x7f766c0fc090 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c111e70 con 0x7f766c0fc090 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f765400f040 con 0x7f766c0fc090 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.953+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76540094b0 con 0x7f766c0fc090 2024-01-19T06:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.954+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c1123e0 con 0x7f766c0fc090 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.954+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c105800 con 0x7f766c0fc090 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.955+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f7654009720 con 0x7f766c0fc090 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.955+0000 7f76637fe640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 0x7f765806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.956+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7654086ae0 con 0x7f766c0fc090 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.956+0000 7f766affd640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 0x7f765806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.956+0000 7f766affd640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 0x7f765806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c009da0 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:15.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:15.960+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7654052f90 con 0x7f766c0fc090 2024-01-19T06:25:16.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.159+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f766c02a720 con 0x7f766c0fc090 2024-01-19T06:25:16.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.160+0000 7f76637fe640 1 -- 172.21.15.134:0/789324119 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4697 (secure 0 0 0) 0x7f76540565b0 con 0x7f766c0fc090 2024-01-19T06:25:16.160 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-19T06:25:16.160 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi134.chvejw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6829/3807665507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":3807665507},{"type":"v1","addr":"172.21.15.134:6829","nonce":3807665507}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":14486,"name":"cephfs.smithi134.higalz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1697349819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1697349819},{"type":"v1","addr":"172.21.15.134:6827","nonce":1697349819}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24267,"name":"cephfs.smithi150.tsxxhg","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.150:6827/2769186085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2769186085},{"type":"v1","addr":"172.21.15.150:6827","nonce":2769186085}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-19T06:24:45.276806+0000","modified":"2024-01-19T06:25:03.694773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi150.xqfgpx","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6825/379459445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":379459445},{"type":"v1","addr":"172.21.15.150:6825","nonce":379459445}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-01-19T06:25:16.160 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 14 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.161+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 msgr2=0x7f765806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.161+0000 7f767199d640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 0x7f765806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f765c000c00 tx=0x7f765c009da0 comp rx=0 tx=0).stop 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.161+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 msgr2=0x7f766c1115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.161+0000 7f767199d640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f765400eb70 tx=0x7f765400b040 comp rx=0 tx=0).stop 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 shutdown_connections 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7658068ed0 0x7f765806b380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f766c0fc090 0x7f766c1115c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 --2- 172.21.15.134:0/789324119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f766c0fae70 0x7f766c065e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 >> 172.21.15.134:0/789324119 conn(0x7f766c0f65c0 msgr2=0x7f766c0f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 shutdown_connections 2024-01-19T06:25:16.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:16.162+0000 7f767199d640 1 -- 172.21.15.134:0/789324119 wait complete. 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/2007982797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/1932630483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:16 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/4127839477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.173 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 1, 'flags': 18} 2024-01-19T06:25:16.174 INFO:teuthology.run_tasks:Running task kclient... 2024-01-19T06:25:16.198 INFO:tasks.kclient:Mounting kernel clients... 2024-01-19T06:25:16.198 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-01-19T06:25:16.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:16.198 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-01-19T06:25:16.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:16.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link delete ceph-brx 2024-01-19T06:25:16.283 INFO:teuthology.orchestra.run.smithi134.stderr:Cannot find device "ceph-brx" 2024-01-19T06:25:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:25:16.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:16.285 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-01-19T06:25:16.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:16.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/2007982797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/1932630483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:25:16.365 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:16 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/4127839477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-19T06:25:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-01-19T06:25:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:25:16.373 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-19T06:25:16.373 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-19T06:25:16.373 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:16.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph fs ls 2024-01-19T06:25:16.628 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:17 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/789324119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T06:25:17.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:17 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/789324119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T06:25:18.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 -- 172.21.15.134:0/2529433466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b200fbb30 msgr2=0x7f8b200fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:18.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 --2- 172.21.15.134:0/2529433466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b200fbb30 0x7f8b200fbfa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b08004940 tx=0x7f8b08009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:18.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 -- 172.21.15.134:0/2529433466 shutdown_connections 2024-01-19T06:25:18.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 --2- 172.21.15.134:0/2529433466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b200fbb30 0x7f8b200fbfa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 --2- 172.21.15.134:0/2529433466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b20101af0 0x7f8b20101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 -- 172.21.15.134:0/2529433466 >> 172.21.15.134:0/2529433466 conn(0x7f8b200f7690 msgr2=0x7f8b200f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 -- 172.21.15.134:0/2529433466 shutdown_connections 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.270+0000 7f8b257a5700 1 -- 172.21.15.134:0/2529433466 wait complete. 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 Processor -- start 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 -- start start 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b200fbb30 0x7f8b2010e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2010f190 con 0x7f8b20101af0 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b257a5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2010f2d0 con 0x7f8b200fbb30 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:18.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35322/0 (socket says 172.21.15.134:35322) 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 -- 172.21.15.134:0/3728399005 learned_addr learned my addr 172.21.15.134:0/3728399005 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b1effd700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b200fbb30 0x7f8b2010e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 -- 172.21.15.134:0/3728399005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b200fbb30 msgr2=0x7f8b2010e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b200fbb30 0x7f8b2010e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.271+0000 7f8b167fc700 1 -- 172.21.15.134:0/3728399005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08005040 con 0x7f8b20101af0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b167fc700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800b3b0 tx=0x7f8b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b0800e040 con 0x7f8b20101af0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b201175d0 con 0x7f8b20101af0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b0801c430 con 0x7f8b20101af0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b08013640 con 0x7f8b20101af0 2024-01-19T06:25:18.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.272+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b20117ae0 con 0x7f8b20101af0 2024-01-19T06:25:18.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.273+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b20103fa0 con 0x7f8b20101af0 2024-01-19T06:25:18.274 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:18 smithi134 ceph-mon[28283]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.273+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f8b0801c5a0 con 0x7f8b20101af0 2024-01-19T06:25:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.273+0000 7f8b17fff700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 0x7f8b0c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.274+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8b08087930 con 0x7f8b20101af0 2024-01-19T06:25:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.274+0000 7f8b1effd700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 0x7f8b0c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.274+0000 7f8b1effd700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 0x7f8b0c06b410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b10004f40 tx=0x7f8b1000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:18.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.279+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8b08053e60 con 0x7f8b20101af0 2024-01-19T06:25:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:18 smithi150 ceph-mon[35328]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T06:25:18.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.509+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8b20064800 con 0x7f8b20101af0 2024-01-19T06:25:18.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.509+0000 7f8b17fff700 1 -- 172.21.15.134:0/3728399005 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f8b08057480 con 0x7f8b20101af0 2024-01-19T06:25:18.509 INFO:teuthology.orchestra.run.smithi134.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-19T06:25:18.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 msgr2=0x7f8b0c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:18.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 0x7f8b0c06b410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b10004f40 tx=0x7f8b1000b040 comp rx=0 tx=0).stop 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 msgr2=0x7f8b2010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800b3b0 tx=0x7f8b08009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 shutdown_connections 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f8b0c068f60 0x7f8b0c06b410 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b200fbb30 0x7f8b2010e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 --2- 172.21.15.134:0/3728399005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b20101af0 0x7f8b2010ec00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.511+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 >> 172.21.15.134:0/3728399005 conn(0x7f8b200f7690 msgr2=0x7f8b200f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.512+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 shutdown_connections 2024-01-19T06:25:18.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:18.512+0000 7f8b257a5700 1 -- 172.21.15.134:0/3728399005 wait complete. 2024-01-19T06:25:19.175 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi134.front.sepia.ceph.com 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-19T06:25:19.176 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-19T06:25:19.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.176 DEBUG:teuthology.orchestra.run.smithi134:> ip addr 2024-01-19T06:25:19.191 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-19T06:25:19.191 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-19T06:25:19.191 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2024-01-19T06:25:19.191 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f0 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f1 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:19.192 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-19T06:25:19.193 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 42277sec preferred_lft 42277sec 2024-01-19T06:25:19.193 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-01-19T06:25:19.193 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:19.193 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-19T06:25:19.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.193 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2024-01-19T06:25:19.193 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-01-19T06:25:19.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add name ceph-brx type bridge 2024-01-19T06:25:19.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr flush dev ceph-brx 2024-01-19T06:25:19.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set ceph-brx up 2024-01-19T06:25:19.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-19T06:25:19.194 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-01-19T06:25:19.312 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:19 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3728399005' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T06:25:19.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.355 DEBUG:teuthology.orchestra.run.smithi134:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-19T06:25:19.381 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-01-19T06:25:19.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.383 DEBUG:teuthology.orchestra.run.smithi134:> route 2024-01-19T06:25:19.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:19 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3728399005' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T06:25:19.454 INFO:teuthology.orchestra.run.smithi134.stdout:Kernel IP routing table 2024-01-19T06:25:19.454 INFO:teuthology.orchestra.run.smithi134.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-19T06:25:19.454 INFO:teuthology.orchestra.run.smithi134.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-19T06:25:19.454 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-19T06:25:19.454 INFO:teuthology.orchestra.run.smithi134.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-19T06:25:19.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-19T06:25:19.455 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-01-19T06:25:19.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.742 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-01-19T06:25:19.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.757 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list-id 2024-01-19T06:25:19.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.812 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2024-01-19T06:25:19.812 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-01-19T06:25:19.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-19T06:25:19.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-19T06:25:19.812 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-01-19T06:25:19.894 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-19T06:25:19.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-19T06:25:19.895 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-01-19T06:25:20.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2024-01-19T06:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-01-19T06:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set brx.0 up 2024-01-19T06:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set dev brx.0 master ceph-brx 2024-01-19T06:25:20.033 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-01-19T06:25:20.074 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T06:25:20.074 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:20.128 INFO:teuthology.orchestra.run.smithi134.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-19T06:25:20.128 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T06:25:20.128 DEBUG:teuthology.orchestra.run.smithi134:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:20.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:20 smithi134 ceph-mon[28283]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-01-19T06:25:20.182 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-19T06:25:20.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T06:25:20.288 INFO:teuthology.orchestra.run.smithi134.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T06:25:20.288 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T06:25:20.288 INFO:teuthology.orchestra.run.smithi134.stdout:invalid new device string format 2024-01-19T06:25:20.398 INFO:teuthology.orchestra.run.smithi134.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T06:25:20.398 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T06:25:20.398 INFO:teuthology.orchestra.run.smithi134.stdout:invalid new device string format 2024-01-19T06:25:20.399 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: resolved to: "172.21.15.134:3300,172.21.15.150:3300" 2024-01-19T06:25:20.399 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:invalid new device string format 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: resolved to: "172.21.15.134:3300,172.21.15.150:3300" 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: resolved to: "172.21.15.134:6789,172.21.15.150:6789" 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.134:6789,172.21.15.150:6789:/ 2024-01-19T06:25:20.404 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=573cd99a-b692-11ee-95ae-87774f69a715" will pass to kernel 2024-01-19T06:25:20.405 INFO:teuthology.orchestra.run.smithi134.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-01-19T06:25:20.405 INFO:teuthology.orchestra.run.smithi134.stdout: You just mounted a file system that supports labels which does not 2024-01-19T06:25:20.405 INFO:teuthology.orchestra.run.smithi134.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-19T06:25:20.405 INFO:teuthology.orchestra.run.smithi134.stdout: applications will generate AVC messages and not be allowed access to 2024-01-19T06:25:20.405 INFO:teuthology.orchestra.run.smithi134.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-19T06:25:20.406 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-19T06:25:20.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:20.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:20 smithi150 ceph-mon[35328]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-01-19T06:25:20.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.435 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:20.500 INFO:teuthology.orchestra.run.smithi134.stdout:client24305 2024-01-19T06:25:20.500 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:25:20.500 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:25:20.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.557 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:20.620 INFO:teuthology.orchestra.run.smithi134.stdout:client24305 2024-01-19T06:25:20.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.620 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:20.683 INFO:teuthology.orchestra.run.smithi134.stdout:client24305 2024-01-19T06:25:20.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:20.683 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/573cd99a-b692-11ee-95ae-87774f69a715.client24305/status') 2024-01-19T06:25:20.756 INFO:teuthology.orchestra.run.smithi134.stdout:instance: client.24305 (3)172.21.15.134:0/3423962089 2024-01-19T06:25:20.756 INFO:teuthology.orchestra.run.smithi134.stdout:blocklisted: false 2024-01-19T06:25:20.756 INFO:teuthology.orchestra.run.smithi134.stderr:0+1 records in 2024-01-19T06:25:20.756 INFO:teuthology.orchestra.run.smithi134.stderr:0+1 records out 2024-01-19T06:25:20.756 INFO:teuthology.orchestra.run.smithi134.stderr:72 bytes copied, 2.0669e-05 s, 3.5 MB/s 2024-01-19T06:25:20.757 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-19T06:25:20.758 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-19T06:25:20.758 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:20.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -- ceph fs ls 2024-01-19T06:25:21.056 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:22.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:22 smithi134 ceph-mon[28283]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:22.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:22 smithi150 ceph-mon[35328]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:22.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 -- 172.21.15.134:0/3899024112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7800101af0 msgr2=0x7f7800101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 --2- 172.21.15.134:0/3899024112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7800101af0 0x7f7800101ec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009ce0 tx=0x7f77f0004d10 comp rx=0 tx=0).stop 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 -- 172.21.15.134:0/3899024112 shutdown_connections 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 --2- 172.21.15.134:0/3899024112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78000fbb30 0x7f78000fbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 --2- 172.21.15.134:0/3899024112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7800101af0 0x7f7800101ec0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 -- 172.21.15.134:0/3899024112 >> 172.21.15.134:0/3899024112 conn(0x7f78000f7670 msgr2=0x7f78000f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 -- 172.21.15.134:0/3899024112 shutdown_connections 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 -- 172.21.15.134:0/3899024112 wait complete. 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.720+0000 7f7806b7c700 1 Processor -- start 2024-01-19T06:25:22.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7806b7c700 1 -- start start 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7806b7c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7806b7c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800101af0 0x7f78000fe060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7806b7c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000fe6f0 con 0x7f78000fbb30 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7806b7c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78000fe860 con 0x7f7800101af0 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35346/0 (socket says 172.21.15.134:35346) 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 -- 172.21.15.134:0/701581739 learned_addr learned my addr 172.21.15.134:0/701581739 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f77fffff700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800101af0 0x7f78000fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 -- 172.21.15.134:0/701581739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800101af0 msgr2=0x7f78000fe060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800101af0 0x7f78000fe060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 -- 172.21.15.134:0/701581739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0009940 con 0x7f78000fbb30 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.721+0000 7f7804918700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f77f000bee0 tx=0x7f77f0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.722+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f000e040 con 0x7f78000fbb30 2024-01-19T06:25:22.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.722+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77f0008f00 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.722+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7800119a50 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.722+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f001c710 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.722+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7800119f40 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.723+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78000fb290 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.723+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f77f0012410 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.723+0000 7f77fd7fa700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 0x7f77e806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.724+0000 7f77fffff700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 0x7f77e806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.724+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f77f00889e0 con 0x7f78000fbb30 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.724+0000 7f77fffff700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 0x7f77e806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f77f4006fd0 tx=0x7f77f4008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:22.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.728+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f77f0054e60 con 0x7f78000fbb30 2024-01-19T06:25:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.954+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f78000ff260 con 0x7f78000fbb30 2024-01-19T06:25:22.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.955+0000 7f77fd7fa700 1 -- 172.21.15.134:0/701581739 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+83 (secure 0 0 0) 0x7f77f0058480 con 0x7f78000fbb30 2024-01-19T06:25:22.955 INFO:teuthology.orchestra.run.smithi134.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 msgr2=0x7f77e806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 0x7f77e806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f77f4006fd0 tx=0x7f77f4008040 comp rx=0 tx=0).stop 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 msgr2=0x7f78000fdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f77f000bee0 tx=0x7f77f0004d10 comp rx=0 tx=0).stop 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 shutdown_connections 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f77e8068f60 0x7f77e806b410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78000fbb30 0x7f78000fdb20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 --2- 172.21.15.134:0/701581739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800101af0 0x7f78000fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 >> 172.21.15.134:0/701581739 conn(0x7f78000f7670 msgr2=0x7f78000ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 shutdown_connections 2024-01-19T06:25:22.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:22.957+0000 7f7806b7c700 1 -- 172.21.15.134:0/701581739 wait complete. 2024-01-19T06:25:23.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:23 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/701581739' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T06:25:23.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:23 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/701581739' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T06:25:23.628 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-19T06:25:23.628 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-19T06:25:23.629 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-19T06:25:23.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:23.629 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-01-19T06:25:23.648 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-19T06:25:23.648 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:23.649 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42285sec preferred_lft 42285sec 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-01-19T06:25:23.650 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-19T06:25:23.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:23.650 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-01-19T06:25:23.650 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-01-19T06:25:23.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-01-19T06:25:23.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-01-19T06:25:23.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-01-19T06:25:23.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-19T06:25:23.651 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-01-19T06:25:23.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:23.897 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-19T06:25:23.926 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-01-19T06:25:23.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:23.928 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-01-19T06:25:24.003 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-01-19T06:25:24.003 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-19T06:25:24.003 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-19T06:25:24.003 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-19T06:25:24.003 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-19T06:25:24.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-19T06:25:24.004 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-01-19T06:25:24.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.251 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-01-19T06:25:24.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.267 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-01-19T06:25:24.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.321 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-01-19T06:25:24.321 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-01-19T06:25:24.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-19T06:25:24.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-19T06:25:24.322 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-01-19T06:25:24.386 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:24 smithi150 ceph-mon[35328]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:24.405 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-19T06:25:24.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.405 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-01-19T06:25:24.405 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-01-19T06:25:24.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-19T06:25:24.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-19T06:25:24.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-19T06:25:24.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-19T06:25:24.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-19T06:25:24.406 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-01-19T06:25:24.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:24 smithi134 ceph-mon[28283]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:24.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.548 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec bash -c ' 2024-01-19T06:25:24.548 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-01-19T06:25:24.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-01-19T06:25:24.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-01-19T06:25:24.548 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-01-19T06:25:24.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T06:25:24.590 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:24.649 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-19T06:25:24.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T06:25:24.649 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:24.703 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-19T06:25:24.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T06:25:24.792 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T06:25:24.793 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T06:25:24.793 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2024-01-19T06:25:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T06:25:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T06:25:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2024-01-19T06:25:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.134:3300,172.21.15.150:3300" 2024-01-19T06:25:24.916 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:invalid new device string format 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.134:3300,172.21.15.150:3300" 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.134:6789,172.21.15.150:6789" 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.134:6789,172.21.15.150:6789:/ 2024-01-19T06:25:24.921 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=573cd99a-b692-11ee-95ae-87774f69a715" will pass to kernel 2024-01-19T06:25:24.922 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-01-19T06:25:24.922 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-01-19T06:25:24.922 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-19T06:25:24.922 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-01-19T06:25:24.922 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-19T06:25:24.924 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-19T06:25:24.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.924 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:24.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:24.952 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:25.008 INFO:teuthology.orchestra.run.smithi150.stdout:client14528 2024-01-19T06:25:25.008 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:25:25.008 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:25:25.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:25.065 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:25.121 INFO:teuthology.orchestra.run.smithi150.stdout:client14528 2024-01-19T06:25:25.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:25.122 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T06:25:25.177 INFO:teuthology.orchestra.run.smithi150.stdout:client14528 2024-01-19T06:25:25.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:25:25.177 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/573cd99a-b692-11ee-95ae-87774f69a715.client14528/status') 2024-01-19T06:25:25.244 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.14528 (3)172.21.15.150:0/2361976056 2024-01-19T06:25:25.244 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2024-01-19T06:25:25.245 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2024-01-19T06:25:25.245 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2024-01-19T06:25:25.245 INFO:teuthology.orchestra.run.smithi150.stderr:72 bytes copied, 2.0995e-05 s, 3.4 MB/s 2024-01-19T06:25:25.246 INFO:teuthology.run_tasks:Running task print... 2024-01-19T06:25:25.257 INFO:teuthology.task.print:**** done client 2024-01-19T06:25:25.257 INFO:teuthology.run_tasks:Running task parallel... 2024-01-19T06:25:25.267 INFO:teuthology.task.parallel:starting parallel... 2024-01-19T06:25:25.268 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-19T06:25:25.268 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-01-19T06:25:25.269 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:25:25.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-01-19T06:25:25.269 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-19T06:25:25.269 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-01-19T06:25:25.273 INFO:tasks.workunit:Pulling workunits from ref 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:25:25.273 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-19T06:25:25.273 INFO:tasks.workunit:timeout=3h 2024-01-19T06:25:25.273 INFO:tasks.workunit:cleanup=True 2024-01-19T06:25:25.273 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:25.295 INFO:teuthology.orchestra.run.smithi134.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:25.295 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-01-19T06:25:25.295 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 72h/114d Inode: 1 Links: 2 2024-01-19T06:25:25.295 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:25:25.296 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-19T06:25:25.296 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-19T06:25:25.296 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-19 06:24:50.305888969 +0000 2024-01-19T06:25:25.296 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-19 06:25:20.432999393 +0000 2024-01-19T06:25:25.296 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-19 06:24:50.305888969 +0000 2024-01-19T06:25:25.296 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-01-19T06:25:25.296 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-01-19T06:25:25.368 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-19T06:25:25.384 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-01-19 06:25:25.363887109 +0000 2024-01-19T06:25:25.385 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-01-19 06:25:25.363887109 +0000 2024-01-19T06:25:25.385 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-01-19 06:24:50.305888969 +0000 2024-01-19T06:25:25.385 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-01-19T06:25:25.385 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-01-19T06:25:25.455 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:25:25.456 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:25:25.472 INFO:tasks.workunit.client.0.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-19T06:25:25.511 INFO:tasks.workunit.client.1.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-01-19T06:25:25.554 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:26.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:26 smithi134 ceph-mon[28283]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:26 smithi150 ceph-mon[35328]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- 172.21.15.134:0/1261933731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 msgr2=0x7f1248101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 --2- 172.21.15.134:0/1261933731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f1248101ba0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f123c004940 tx=0x7f123c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- 172.21.15.134:0/1261933731 shutdown_connections 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 --2- 172.21.15.134:0/1261933731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 0x7f12481044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 --2- 172.21.15.134:0/1261933731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f1248101ba0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- 172.21.15.134:0/1261933731 >> 172.21.15.134:0/1261933731 conn(0x7f12480f71b0 msgr2=0x7f12480f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- 172.21.15.134:0/1261933731 shutdown_connections 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- 172.21.15.134:0/1261933731 wait complete. 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 Processor -- start 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.330+0000 7f124ebac700 1 -- start start 2024-01-19T06:25:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124ebac700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124ebac700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 0x7f124810fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124ebac700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1248110330 con 0x7f1248066920 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124ebac700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1248110b70 con 0x7f12481020e0 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37288/0 (socket says 172.21.15.134:37288) 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 -- 172.21.15.134:0/3147133145 learned_addr learned my addr 172.21.15.134:0/3147133145 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 -- 172.21.15.134:0/3147133145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 msgr2=0x7f124810fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f1247fff700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 0x7f124810fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 0x7f124810fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 -- 172.21.15.134:0/3147133145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c005040 con 0x7f1248066920 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124c948700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f123c009380 tx=0x7f123c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c00e040 con 0x7f1248066920 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f123c00bda0 con 0x7f1248066920 2024-01-19T06:25:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.331+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1248110df0 con 0x7f1248066920 2024-01-19T06:25:27.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.332+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c01c630 con 0x7f1248066920 2024-01-19T06:25:27.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.332+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1248111330 con 0x7f1248066920 2024-01-19T06:25:27.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.332+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124804bb80 con 0x7f1248066920 2024-01-19T06:25:27.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.333+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f123c0139d0 con 0x7f1248066920 2024-01-19T06:25:27.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.333+0000 7f12457fa700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 0x7f123806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:27.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.333+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f123c0882b0 con 0x7f1248066920 2024-01-19T06:25:27.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.333+0000 7f1247fff700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 0x7f123806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:27.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.334+0000 7f1247fff700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 0x7f123806b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1234006fd0 tx=0x7f1234008040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:27.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.338+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f123c0581c0 con 0x7f1248066920 2024-01-19T06:25:27.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.523+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f1248107820 con 0x7f1248066920 2024-01-19T06:25:27.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.523+0000 7f12457fa700 1 -- 172.21.15.134:0/3147133145 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f123c057d50 con 0x7f1248066920 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.525+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 msgr2=0x7f123806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.525+0000 7f124ebac700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 0x7f123806b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1234006fd0 tx=0x7f1234008040 comp rx=0 tx=0).stop 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.525+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 msgr2=0x7f124810f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.525+0000 7f124ebac700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f123c009380 tx=0x7f123c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 shutdown_connections 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f1238068f60 0x7f123806b410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1248066920 0x7f124810f760 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 --2- 172.21.15.134:0/3147133145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12481020e0 0x7f124810fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:27.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 >> 172.21.15.134:0/3147133145 conn(0x7f12480f71b0 msgr2=0x7f12480f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:27.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 shutdown_connections 2024-01-19T06:25:27.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:27.526+0000 7f124ebac700 1 -- 172.21.15.134:0/3147133145 wait complete. 2024-01-19T06:25:28.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:28 smithi134 ceph-mon[28283]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:28.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-01-19T06:25:28.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:28 smithi150 ceph-mon[35328]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:25:28.502 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:30.268 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:30 smithi134 ceph-mon[28283]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- 172.21.15.134:0/2330627238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 msgr2=0x7fbe0c06ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 --2- 172.21.15.134:0/2330627238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c06ffb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc004940 tx=0x7fbdfc009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- 172.21.15.134:0/2330627238 shutdown_connections 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 --2- 172.21.15.134:0/2330627238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 --2- 172.21.15.134:0/2330627238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c06ffb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- 172.21.15.134:0/2330627238 >> 172.21.15.134:0/2330627238 conn(0x7fbe0c0f82d0 msgr2=0x7fbe0c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- 172.21.15.134:0/2330627238 shutdown_connections 2024-01-19T06:25:30.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- 172.21.15.134:0/2330627238 wait complete. 2024-01-19T06:25:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 Processor -- start 2024-01-19T06:25:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- start start 2024-01-19T06:25:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c10f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:30.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c110350 con 0x7fbe0c071b40 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.272+0000 7fbe1088f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c110bf0 con 0x7fbe0c0704f0 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe0a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c10f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37302/0 (socket says 172.21.15.134:37302) 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe0a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c10f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:34536/0 (socket says 172.21.15.134:34536) 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 -- 172.21.15.134:0/3380782970 learned_addr learned my addr 172.21.15.134:0/3380782970 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 -- 172.21.15.134:0/3380782970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 msgr2=0x7fbe0c10f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c10f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 -- 172.21.15.134:0/3380782970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc005040 con 0x7fbe0c071b40 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe09d9b700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000eb40 tx=0x7fbe0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe00007cc0 con 0x7fbe0c071b40 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe0000f040 con 0x7fbe0c071b40 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c110e70 con 0x7fbe0c071b40 2024-01-19T06:25:30.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe0000c990 con 0x7fbe0c071b40 2024-01-19T06:25:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.273+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c111360 con 0x7fbe0c071b40 2024-01-19T06:25:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.275+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c1096e0 con 0x7fbe0c071b40 2024-01-19T06:25:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.275+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7fbe0000cc00 con 0x7fbe0c071b40 2024-01-19T06:25:30.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.275+0000 7fbdfaffd700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 0x7fbdf406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:30.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.275+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbe000869f0 con 0x7fbe0c071b40 2024-01-19T06:25:30.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.275+0000 7fbe0a59c700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 0x7fbdf406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:30.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.276+0000 7fbe0a59c700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 0x7fbdf406b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:30.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.280+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbe0004da00 con 0x7fbe0c071b40 2024-01-19T06:25:30.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:30 smithi150 ceph-mon[35328]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:30.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.469+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fbe0c1078f0 con 0x7fbe0c071b40 2024-01-19T06:25:30.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.469+0000 7fbdfaffd700 1 -- 172.21.15.134:0/3380782970 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fbe00088e60 con 0x7fbe0c071b40 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 msgr2=0x7fbdf406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 0x7fbdf406b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc000c00 tx=0x7fbdfc009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 msgr2=0x7fbe0c10fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000eb40 tx=0x7fbe0000b040 comp rx=0 tx=0).stop 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 shutdown_connections 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7fbdf4068f60 0x7fbdf406b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe0c0704f0 0x7fbe0c10f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 --2- 172.21.15.134:0/3380782970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c071b40 0x7fbe0c10fd50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:30.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 >> 172.21.15.134:0/3380782970 conn(0x7fbe0c0f82d0 msgr2=0x7fbe0c103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:30.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 shutdown_connections 2024-01-19T06:25:30.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:30.472+0000 7fbe1088f700 1 -- 172.21.15.134:0/3380782970 wait complete. 2024-01-19T06:25:31.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set global log_to_journald false --force' 2024-01-19T06:25:31.453 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:32.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:32 smithi134 ceph-mon[28283]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:32.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:32 smithi150 ceph-mon[35328]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.196+0000 7f3d50b12700 1 -- 172.21.15.134:0/3342070930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 msgr2=0x7f3d4c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.196+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3342070930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c0fbf00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c004940 tx=0x7f3d3c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 -- 172.21.15.134:0/3342070930 shutdown_connections 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3342070930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3342070930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c0fbf00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 -- 172.21.15.134:0/3342070930 >> 172.21.15.134:0/3342070930 conn(0x7f3d4c0f7190 msgr2=0x7f3d4c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:33.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 -- 172.21.15.134:0/3342070930 shutdown_connections 2024-01-19T06:25:33.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 -- 172.21.15.134:0/3342070930 wait complete. 2024-01-19T06:25:33.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.197+0000 7f3d50b12700 1 Processor -- start 2024-01-19T06:25:33.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d50b12700 1 -- start start 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d50b12700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d50b12700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c10c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d50b12700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c10cf30 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d50b12700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c10d7d0 con 0x7f3d4c0fcd00 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37318/0 (socket says 172.21.15.134:37318) 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 -- 172.21.15.134:0/3375988366 learned_addr learned my addr 172.21.15.134:0/3375988366 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d49d9b700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c10c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 -- 172.21.15.134:0/3375988366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 msgr2=0x7f3d4c10c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c10c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 -- 172.21.15.134:0/3375988366 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c005040 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d49d9b700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c10c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T06:25:33.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.198+0000 7f3d4a59c700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c004890 tx=0x7f3d3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:33.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.199+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d3c00e070 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.199+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d3c013c40 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.199+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c10da50 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.199+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d3c023630 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.199+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c10df30 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.200+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c04bb80 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.200+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f3d3c02c3f0 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.201+0000 7f3d3affd700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 0x7f3d3406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.201+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3d3c08a4f0 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.201+0000 7f3d49d9b700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 0x7f3d3406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:33.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.201+0000 7f3d49d9b700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 0x7f3d3406b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009b80 tx=0x7f3d4000b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:33.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.205+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3d3c05a5f0 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.413+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f3d4c104490 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.414+0000 7f3d3affd700 1 -- 172.21.15.134:0/3375988366 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f3d3c05a180 con 0x7f3d4c0fbaf0 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 msgr2=0x7f3d3406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 0x7f3d3406b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009b80 tx=0x7f3d4000b040 comp rx=0 tx=0).stop 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 msgr2=0x7f3d4c10c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c004890 tx=0x7f3d3c009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 shutdown_connections 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d34068f60 0x7f3d3406b410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d4c0fbaf0 0x7f3d4c10c3f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 --2- 172.21.15.134:0/3375988366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4c0fcd00 0x7f3d4c10c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 >> 172.21.15.134:0/3375988366 conn(0x7f3d4c0f7190 msgr2=0x7f3d4c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:33.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 shutdown_connections 2024-01-19T06:25:33.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:33.417+0000 7f3d50b12700 1 -- 172.21.15.134:0/3375988366 wait complete. 2024-01-19T06:25:34.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-01-19T06:25:34.405 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:25:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:34 smithi134 ceph-mon[28283]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:34 smithi150 ceph-mon[35328]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T06:25:36.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:36 smithi134 ceph-mon[28283]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.213+0000 7f16930d0700 1 -- 172.21.15.134:0/1605274516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 msgr2=0x7f168c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.213+0000 7f16930d0700 1 --2- 172.21.15.134:0/1605274516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c0fbf00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1680004940 tx=0x7f1680009e30 comp rx=0 tx=0).stop 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- 172.21.15.134:0/1605274516 shutdown_connections 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 --2- 172.21.15.134:0/1605274516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 0x7f168c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 --2- 172.21.15.134:0/1605274516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c0fbf00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- 172.21.15.134:0/1605274516 >> 172.21.15.134:0/1605274516 conn(0x7f168c0f71b0 msgr2=0x7f168c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:25:36.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- 172.21.15.134:0/1605274516 shutdown_connections 2024-01-19T06:25:36.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- 172.21.15.134:0/1605274516 wait complete. 2024-01-19T06:25:36.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 Processor -- start 2024-01-19T06:25:36.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- start start 2024-01-19T06:25:36.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:36.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 0x7f168c10d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:36.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c10dff0 con 0x7f168c0fbaf0 2024-01-19T06:25:36.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.214+0000 7f16930d0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c10e890 con 0x7f168c0fcd00 2024-01-19T06:25:36.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37338/0 (socket says 172.21.15.134:37338) 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f168bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 0x7f168c10d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 -- 172.21.15.134:0/3109784765 learned_addr learned my addr 172.21.15.134:0/3109784765 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 -- 172.21.15.134:0/3109784765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 msgr2=0x7f168c10d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 0x7f168c10d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 -- 172.21.15.134:0/3109784765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f1690e6c700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1680000c00 tx=0x7f1680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168000e040 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1680009af0 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c10eb10 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c10f050 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.215+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168001c630 con 0x7f168c0fbaf0 2024-01-19T06:25:36.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.216+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c04bb80 con 0x7f168c0fbaf0 2024-01-19T06:25:36.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.218+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f168000bc50 con 0x7f168c0fbaf0 2024-01-19T06:25:36.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.218+0000 7f16897fa700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 0x7f167c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:25:36.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.218+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1680088150 con 0x7f168c0fbaf0 2024-01-19T06:25:36.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.218+0000 7f168bfff700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 0x7f167c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:25:36.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.219+0000 7f168bfff700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 0x7f167c06b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1678009fd0 tx=0x7f1678011040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:25:36.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.226+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1680058060 con 0x7f168c0fbaf0 2024-01-19T06:25:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:36 smithi150 ceph-mon[35328]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:36.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:25:36.474+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f168c10f4f0 con 0x7f167c068f60 2024-01-19T06:25:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:38 smithi134 ceph-mon[28283]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:25:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:38 smithi134 ceph-mon[28283]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:38.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:38 smithi150 ceph-mon[35328]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:25:38.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:38 smithi150 ceph-mon[35328]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:40.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:40 smithi134 ceph-mon[28283]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:40.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:40 smithi150 ceph-mon[35328]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:25:41.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:41 smithi134 ceph-mon[28283]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:41.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:41 smithi150 ceph-mon[35328]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:44.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:44 smithi134 ceph-mon[28283]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:44.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:44 smithi150 ceph-mon[35328]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:46.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:46 smithi150 ceph-mon[35328]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:46.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:46 smithi134 ceph-mon[28283]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:48.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:48 smithi134 ceph-mon[28283]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:48.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:48 smithi150 ceph-mon[35328]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:50.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:50 smithi134 ceph-mon[28283]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:50.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:50 smithi150 ceph-mon[35328]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:52 smithi134 ceph-mon[28283]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:52 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:52 smithi150 ceph-mon[35328]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:52.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:52 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:25:54.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:54 smithi134 ceph-mon[28283]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:54.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:54 smithi150 ceph-mon[35328]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:56.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:56 smithi150 ceph-mon[35328]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:56.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:56 smithi134 ceph-mon[28283]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:58.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:25:58 smithi134 ceph-mon[28283]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:25:58.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:25:58 smithi150 ceph-mon[35328]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:26:00.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:00 smithi134 ceph-mon[28283]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:26:00.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:00 smithi150 ceph-mon[35328]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:26:02.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:02 smithi134 ceph-mon[28283]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:26:02.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:02 smithi150 ceph-mon[35328]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T06:26:04.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:04 smithi134 ceph-mon[28283]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-01-19T06:26:04.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:04 smithi150 ceph-mon[35328]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-01-19T06:26:06.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:06 smithi134 ceph-mon[28283]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T06:26:06.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:06 smithi150 ceph-mon[35328]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T06:26:07.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:07 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:07.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:07 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:08.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:08 smithi134 ceph-mon[28283]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T06:26:08.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:08 smithi150 ceph-mon[35328]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T06:26:10.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:10 smithi134 ceph-mon[28283]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:10.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:10 smithi150 ceph-mon[35328]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:state without impacting any branches by switching back to a branch. 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.728 INFO:tasks.workunit.client.1.smithi150.stderr: git switch -c 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr:Or undo this operation with: 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr: git switch - 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr: 2024-01-19T06:26:10.729 INFO:tasks.workunit.client.1.smithi150.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-19T06:26:10.732 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-01-19T06:26:10.796 INFO:tasks.workunit.client.1.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-19T06:26:10.798 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-19T06:26:10.798 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-19T06:26:10.798 INFO:teuthology.orchestra.run.smithi134.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:26:10.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.798+0000 7f16897fa700 1 -- 172.21.15.134:0/3109784765 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f168c10f4f0 con 0x7f167c068f60 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 msgr2=0x7f167c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 0x7f167c06b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1678009fd0 tx=0x7f1678011040 comp rx=0 tx=0).stop 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 msgr2=0x7f168c10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1680000c00 tx=0x7f1680009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 shutdown_connections 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f167c068f60 0x7f167c06b410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f168c0fbaf0 0x7f168c10d4b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 --2- 172.21.15.134:0/3109784765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c0fcd00 0x7f168c10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:10.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 >> 172.21.15.134:0/3109784765 conn(0x7f168c0f71b0 msgr2=0x7f168c065be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:10.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 shutdown_connections 2024-01-19T06:26:10.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:10.801+0000 7f16930d0700 1 -- 172.21.15.134:0/3109784765 wait complete. 2024-01-19T06:26:11.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-01-19T06:26:11.578 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-19T06:26:11.643 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-19T06:26:11.683 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-19T06:26:11.684 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-19T06:26:11.684 INFO:tasks.workunit.client.1.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-19T06:26:11.722 INFO:tasks.workunit.client.1.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-19T06:26:11.725 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:26:11.725 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-01-19T06:26:11.782 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-01-19T06:26:11.784 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-19T06:26:11.784 DEBUG:teuthology.orchestra.run.smithi150:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-01-19T06:26:11.842 INFO:tasks.workunit.client.1.smithi150.stderr:+ mkdir -p fsstress 2024-01-19T06:26:11.843 INFO:tasks.workunit.client.1.smithi150.stderr:+ pushd fsstress 2024-01-19T06:26:11.844 INFO:tasks.workunit.client.1.smithi150.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-19T06:26:11.844 INFO:tasks.workunit.client.1.smithi150.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-19T06:26:11.880 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:26:12.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:11 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:12.182 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T06:26:12.183 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:11 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:12.355 INFO:tasks.workunit.client.1.smithi150.stderr:+ tar xzf ltp-full.tgz 2024-01-19T06:26:13.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:13 smithi134 ceph-mon[28283]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:26:13.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:13 smithi150 ceph-mon[35328]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T06:26:13.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 -- 172.21.15.134:0/1221277279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c40fde00 msgr2=0x7f44c40fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:13.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/1221277279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c40fde00 0x7f44c40fe250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f44b4004940 tx=0x7f44b4009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:13.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 -- 172.21.15.134:0/1221277279 shutdown_connections 2024-01-19T06:26:13.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/1221277279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c40fde00 0x7f44c40fe250 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:13.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/1221277279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c40fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:13.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.803+0000 7f44cb2e9700 1 -- 172.21.15.134:0/1221277279 >> 172.21.15.134:0/1221277279 conn(0x7f44c40f82d0 msgr2=0x7f44c40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:13.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.804+0000 7f44cb2e9700 1 -- 172.21.15.134:0/1221277279 shutdown_connections 2024-01-19T06:26:13.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.804+0000 7f44cb2e9700 1 -- 172.21.15.134:0/1221277279 wait complete. 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 Processor -- start 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 -- start start 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c410e050 con 0x7f44c410db90 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44cb2e9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c410e190 con 0x7f44c40fcbf0 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44c9085700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44c8884700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44c9085700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:60082/0 (socket says 172.21.15.134:60082) 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44c8884700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40416/0 (socket says 172.21.15.134:40416) 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.805+0000 7f44c9085700 1 -- 172.21.15.134:0/2812761152 learned_addr learned my addr 172.21.15.134:0/2812761152 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44c9085700 1 -- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 msgr2=0x7f44c410ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44c9085700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44c9085700 1 -- 172.21.15.134:0/2812761152 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b4005040 con 0x7f44c40fcbf0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44c8884700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44c9085700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44c0010e80 tx=0x7f44c000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c0015070 con 0x7f44c40fcbf0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44c000bde0 con 0x7f44c40fcbf0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c410f030 con 0x7f44c40fcbf0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.806+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c001b470 con 0x7f44c40fcbf0 2024-01-19T06:26:13.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.807+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c410f540 con 0x7f44c40fcbf0 2024-01-19T06:26:13.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.808+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c40fbf50 con 0x7f44c40fcbf0 2024-01-19T06:26:13.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.808+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f44c00091d0 con 0x7f44c40fcbf0 2024-01-19T06:26:13.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.809+0000 7f44b9ffb700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 0x7f44b006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:13.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.809+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f44c008afb0 con 0x7f44c40fcbf0 2024-01-19T06:26:13.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.809+0000 7f44c8884700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 0x7f44b006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:13.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.809+0000 7f44c8884700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 0x7f44b006b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44b400b0d0 tx=0x7f44b4009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:13.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:13.817+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f44c005aec0 con 0x7f44c40fcbf0 2024-01-19T06:26:14.127 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:14 smithi134 ceph-mon[28283]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:14.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.152+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44c41027f0 con 0x7f44b0068f70 2024-01-19T06:26:14.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.154+0000 7f44b9ffb700 1 -- 172.21.15.134:0/2812761152 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f44c41027f0 con 0x7f44b0068f70 2024-01-19T06:26:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.157+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 msgr2=0x7f44b006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.157+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 0x7f44b006b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f44b400b0d0 tx=0x7f44b4009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.157+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 msgr2=0x7f44c410d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.157+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44c0010e80 tx=0x7f44c000b040 comp rx=0 tx=0).stop 2024-01-19T06:26:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 shutdown_connections 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f44b0068f70 0x7f44b006b420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44c40fcbf0 0x7f44c410d650 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 --2- 172.21.15.134:0/2812761152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c410db90 0x7f44c410ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 >> 172.21.15.134:0/2812761152 conn(0x7f44c40f82d0 msgr2=0x7f44c41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 shutdown_connections 2024-01-19T06:26:14.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.158+0000 7f44cb2e9700 1 -- 172.21.15.134:0/2812761152 wait complete. 2024-01-19T06:26:14.174 INFO:teuthology.orchestra.run.smithi134.stdout:true 2024-01-19T06:26:14.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.326+0000 7f7f9b59e700 1 -- 172.21.15.134:0/3692588081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 msgr2=0x7f7f9409c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.326+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/3692588081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f9409c860 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004940 tx=0x7f7f90009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:14.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 -- 172.21.15.134:0/3692588081 shutdown_connections 2024-01-19T06:26:14.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/3692588081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f9409c860 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/3692588081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f9409b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 -- 172.21.15.134:0/3692588081 >> 172.21.15.134:0/3692588081 conn(0x7f7f9400a500 msgr2=0x7f7f94099350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:14.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 -- 172.21.15.134:0/3692588081 shutdown_connections 2024-01-19T06:26:14.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.327+0000 7f7f9b59e700 1 -- 172.21.15.134:0/3692588081 wait complete. 2024-01-19T06:26:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.328+0000 7f7f9b59e700 1 Processor -- start 2024-01-19T06:26:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.329+0000 7f7f9b59e700 1 -- start start 2024-01-19T06:26:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.329+0000 7f7f9b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.329+0000 7f7f9b59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f940abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f9a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f9b59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940ac2e0 con 0x7f7f9409c410 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f9a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:60102/0 (socket says 172.21.15.134:60102) 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f9a59c700 1 -- 172.21.15.134:0/1769450909 learned_addr learned my addr 172.21.15.134:0/1769450909 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f99d9b700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f940abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.330+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940ac450 con 0x7f7f9409b200 2024-01-19T06:26:14.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.331+0000 7f7f99d9b700 1 -- 172.21.15.134:0/1769450909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 msgr2=0x7f7f940ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.331+0000 7f7f99d9b700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.331+0000 7f7f99d9b700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f90005040 con 0x7f7f9409c410 2024-01-19T06:26:14.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.331+0000 7f7f9a59c700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T06:26:14.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.332+0000 7f7f99d9b700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f940abce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004890 tx=0x7f7f90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:14.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.332+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9000e070 con 0x7f7f9409c410 2024-01-19T06:26:14.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.332+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f9000bd90 con 0x7f7f9409c410 2024-01-19T06:26:14.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.332+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f90013ea0 con 0x7f7f9409c410 2024-01-19T06:26:14.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.333+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f940b0ea0 con 0x7f7f9409c410 2024-01-19T06:26:14.333 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:14 smithi150 ceph-mon[35328]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T06:26:14.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.333+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f940b1430 con 0x7f7f9409c410 2024-01-19T06:26:14.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.335+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f7f90013420 con 0x7f7f9409c410 2024-01-19T06:26:14.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.336+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f940a3ba0 con 0x7f7f9409c410 2024-01-19T06:26:14.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.336+0000 7f7f8affd700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 0x7f7f8406b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.337+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7f90088640 con 0x7f7f9409c410 2024-01-19T06:26:14.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.340+0000 7f7f9a59c700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 0x7f7f8406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.341+0000 7f7f9a59c700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 0x7f7f8406b530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e5b0 tx=0x7f7f8c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:14.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.345+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7f90058530 con 0x7f7f9409c410 2024-01-19T06:26:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.739+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f940b1730 con 0x7f7f84069080 2024-01-19T06:26:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.740+0000 7f7f8affd700 1 -- 172.21.15.134:0/1769450909 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f7f940b1730 con 0x7f7f84069080 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 msgr2=0x7f7f8406b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 0x7f7f8406b530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00e5b0 tx=0x7f7f8c00b040 comp rx=0 tx=0).stop 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 msgr2=0x7f7f940abce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f940abce0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7f90004890 tx=0x7f7f90009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 shutdown_connections 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7f84069080 0x7f7f8406b530 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f9409b200 0x7f7f940ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 --2- 172.21.15.134:0/1769450909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f9409c410 0x7f7f940abce0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 >> 172.21.15.134:0/1769450909 conn(0x7f7f9400a500 msgr2=0x7f7f9400c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.743+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 shutdown_connections 2024-01-19T06:26:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.744+0000 7f7f9b59e700 1 -- 172.21.15.134:0/1769450909 wait complete. 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 -- 172.21.15.134:0/3592602114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 msgr2=0x7f9f441044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 --2- 172.21.15.134:0/3592602114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f441044c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9f38004940 tx=0x7f9f38009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 -- 172.21.15.134:0/3592602114 shutdown_connections 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 --2- 172.21.15.134:0/3592602114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f441044c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 --2- 172.21.15.134:0/3592602114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f44101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 -- 172.21.15.134:0/3592602114 >> 172.21.15.134:0/3592602114 conn(0x7f9f440f7190 msgr2=0x7f9f440f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 -- 172.21.15.134:0/3592602114 shutdown_connections 2024-01-19T06:26:14.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.854+0000 7f9f4920e700 1 -- 172.21.15.134:0/3592602114 wait complete. 2024-01-19T06:26:14.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.855+0000 7f9f4920e700 1 Processor -- start 2024-01-19T06:26:14.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f4920e700 1 -- start start 2024-01-19T06:26:14.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f4920e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f4410f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f4920e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f42d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f4410f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f42d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f4410f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:60108/0 (socket says 172.21.15.134:60108) 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.856+0000 7f9f4259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40448/0 (socket says 172.21.15.134:40448) 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f42d9d700 1 -- 172.21.15.134:0/4162161985 learned_addr learned my addr 172.21.15.134:0/4162161985 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f44110390 con 0x7f9f441020e0 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f44110c30 con 0x7f9f44066090 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4259c700 1 -- 172.21.15.134:0/4162161985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 msgr2=0x7f9f4410f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4259c700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f4410f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:14.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.857+0000 7f9f4259c700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38005040 con 0x7f9f441020e0 2024-01-19T06:26:14.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.858+0000 7f9f4259c700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009fd0 tx=0x7f9f38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:14.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.858+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f38020070 con 0x7f9f441020e0 2024-01-19T06:26:14.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.858+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f44110eb0 con 0x7f9f441020e0 2024-01-19T06:26:14.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.858+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f44111390 con 0x7f9f441020e0 2024-01-19T06:26:14.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.859+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f3801c400 con 0x7f9f441020e0 2024-01-19T06:26:14.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.859+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f3800e070 con 0x7f9f441020e0 2024-01-19T06:26:14.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.860+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f300052e0 con 0x7f9f441020e0 2024-01-19T06:26:14.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.861+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f9f38013a50 con 0x7f9f441020e0 2024-01-19T06:26:14.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.862+0000 7f9f2b7fe700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 0x7f9f2c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:14.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.862+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9f38088770 con 0x7f9f441020e0 2024-01-19T06:26:14.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.863+0000 7f9f42d9d700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 0x7f9f2c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:14.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.864+0000 7f9f42d9d700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 0x7f9f2c06b420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f34008fd0 tx=0x7f9f3400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:14.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:14.868+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9f38058660 con 0x7f9f441020e0 2024-01-19T06:26:15.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.172+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f30000bf0 con 0x7f9f2c068f70 2024-01-19T06:26:15.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.189+0000 7f9f2b7fe700 1 -- 172.21.15.134:0/4162161985 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f9f30000bf0 con 0x7f9f2c068f70 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:alertmanager.smithi134 smithi134 *:9093,9094 running (5m) 59s ago 6m 24.4M - 0.25.0 c8568f914cd2 09f464d8c526 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi134 smithi134 running (6m) 59s ago 6m 6665k - 17.2.7-268-g6441305d c3a232f785d8 434084e15158 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi150 smithi150 running (5m) 65s ago 5m 6660k - 17.2.7-268-g6441305d c3a232f785d8 1aee22edee9f 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:grafana.smithi134 smithi134 *:3000 running (5m) 59s ago 6m 84.4M - 9.4.7 954c08fa6188 d3ec6951f4ff 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi134.chvejw smithi134 running (78s) 59s ago 78s 11.3M - 17.2.7-268-g6441305d c3a232f785d8 86973986f1f1 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi134.higalz smithi134 running (85s) 59s ago 85s 10.4M - 17.2.7-268-g6441305d c3a232f785d8 86c9152ca3d8 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi150.tsxxhg smithi150 running (76s) 65s ago 75s 13.5M - 17.2.7-268-g6441305d c3a232f785d8 bc505d3ee14e 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi150.xqfgpx smithi150 running (83s) 65s ago 82s 13.3M - 17.2.7-268-g6441305d c3a232f785d8 0c977cab8f03 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi134.qrxokr smithi134 *:9283 running (8m) 59s ago 8m 481M - 17.2.7-268-g6441305d c3a232f785d8 6f69c250d5a7 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi150.xkxrco smithi150 *:8443,9283 running (5m) 65s ago 5m 433M - 17.2.7-268-g6441305d c3a232f785d8 d6d4dd5f44c3 2024-01-19T06:26:15.190 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi134 smithi134 running (8m) 59s ago 8m 45.0M 2048M 17.2.7-268-g6441305d c3a232f785d8 328c6cb50ef7 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi150 smithi150 running (5m) 65s ago 5m 35.6M 2048M 17.2.7-268-g6441305d c3a232f785d8 36884ce908a9 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi134 smithi134 *:9100 running (6m) 59s ago 6m 15.7M - 1.5.0 0da6a335fe13 9ff3509f6e9d 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi150 smithi150 *:9100 running (5m) 65s ago 5m 15.6M - 1.5.0 0da6a335fe13 f5edbad1d4e4 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (5m) 59s ago 5m 49.0M 4288M 17.2.7-268-g6441305d c3a232f785d8 c42a610a3422 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi134 running (4m) 59s ago 4m 47.5M 4288M 17.2.7-268-g6441305d c3a232f785d8 980268e5252b 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi134 running (4m) 59s ago 4m 44.5M 4288M 17.2.7-268-g6441305d c3a232f785d8 cbc144916de2 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi150 running (3m) 65s ago 3m 45.2M 5312M 17.2.7-268-g6441305d c3a232f785d8 799be7524f32 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 smithi150 running (3m) 65s ago 3m 44.2M 5312M 17.2.7-268-g6441305d c3a232f785d8 4e0d71b857f1 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 smithi150 running (2m) 65s ago 2m 40.6M 5312M 17.2.7-268-g6441305d c3a232f785d8 175a03fc551b 2024-01-19T06:26:15.191 INFO:teuthology.orchestra.run.smithi134.stdout:prometheus.smithi134 smithi134 *:9095 running (5m) 59s ago 6m 56.4M - 2.43.0 a07b618ecd1d e5681937937a 2024-01-19T06:26:15.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 msgr2=0x7f9f2c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 0x7f9f2c06b420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f34008fd0 tx=0x7f9f3400b040 comp rx=0 tx=0).stop 2024-01-19T06:26:15.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 msgr2=0x7f9f4410fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009fd0 tx=0x7f9f38009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 shutdown_connections 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9f2c068f70 0x7f9f2c06b420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f44066090 0x7f9f4410f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 --2- 172.21.15.134:0/4162161985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f441020e0 0x7f9f4410fd90 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009fd0 tx=0x7f9f38009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.192+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 >> 172.21.15.134:0/4162161985 conn(0x7f9f440f7190 msgr2=0x7f9f440f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.193+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 shutdown_connections 2024-01-19T06:26:15.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.193+0000 7f9f4920e700 1 -- 172.21.15.134:0/4162161985 wait complete. 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 -- 172.21.15.134:0/3077350773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7968066070 msgr2=0x7f79680ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 --2- 172.21.15.134:0/3077350773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7968066070 0x7f79680ff7c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7958004940 tx=0x7f7958009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 -- 172.21.15.134:0/3077350773 shutdown_connections 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 --2- 172.21.15.134:0/3077350773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79680ffd90 0x7f79681021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 --2- 172.21.15.134:0/3077350773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7968066070 0x7f79680ff7c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.346+0000 7f796cf75700 1 -- 172.21.15.134:0/3077350773 >> 172.21.15.134:0/3077350773 conn(0x7f79680f7190 msgr2=0x7f79680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:15.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.347+0000 7f796cf75700 1 -- 172.21.15.134:0/3077350773 shutdown_connections 2024-01-19T06:26:15.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.347+0000 7f796cf75700 1 -- 172.21.15.134:0/3077350773 wait complete. 2024-01-19T06:26:15.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 Processor -- start 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 -- start start 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 0x7f796810da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796810e000 con 0x7f79680ffd90 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796cf75700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796810e8a0 con 0x7f7968066070 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f7965d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 0x7f796810da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.134:60136/0 (socket says 172.21.15.134:60136) 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f7965d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 0x7f796810da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40464/0 (socket says 172.21.15.134:40464) 2024-01-19T06:26:15.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.348+0000 7f796659c700 1 -- 172.21.15.134:0/3926864797 learned_addr learned my addr 172.21.15.134:0/3926864797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796659c700 1 -- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 msgr2=0x7f796810da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796659c700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 0x7f796810da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796659c700 1 -- 172.21.15.134:0/3926864797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958005040 con 0x7f7968066070 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796659c700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7958008f90 tx=0x7f7958009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7958021070 con 0x7f7968066070 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f795801c400 con 0x7f7968066070 2024-01-19T06:26:15.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7958021070 con 0x7f7968066070 2024-01-19T06:26:15.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796810eb20 con 0x7f7968066070 2024-01-19T06:26:15.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.349+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796810f010 con 0x7f7968066070 2024-01-19T06:26:15.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.351+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79480052e0 con 0x7f7968066070 2024-01-19T06:26:15.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.352+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f7958013a30 con 0x7f7968066070 2024-01-19T06:26:15.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.352+0000 7f7956ffd700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 0x7f795006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:15.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.352+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f795805b200 con 0x7f7968066070 2024-01-19T06:26:15.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.353+0000 7f7965d9b700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 0x7f795006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:15.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.353+0000 7f7965d9b700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 0x7f795006b4d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f795c004600 tx=0x7f795c00b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:15.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.358+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7958054a80 con 0x7f7968066070 2024-01-19T06:26:15.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.863+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7948005860 con 0x7f7968066070 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.864+0000 7f7956ffd700 1 -- 172.21.15.134:0/3926864797 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f7958054a80 con 0x7f7968066070 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: "mon": { 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:15.864 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": { 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": { 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 6 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 4 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "overall": { 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 14 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-01-19T06:26:15.865 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-01-19T06:26:15.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 msgr2=0x7f795006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 0x7f795006b4d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f795c004600 tx=0x7f795c00b040 comp rx=0 tx=0).stop 2024-01-19T06:26:15.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 msgr2=0x7f796810d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:15.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7958008f90 tx=0x7f7958009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 shutdown_connections 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f7950069020 0x7f795006b4d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7968066070 0x7f796810d4c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 --2- 172.21.15.134:0/3926864797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79680ffd90 0x7f796810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.866+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 >> 172.21.15.134:0/3926864797 conn(0x7f79680f7190 msgr2=0x7f79680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.867+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 shutdown_connections 2024-01-19T06:26:15.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:15.867+0000 7f796cf75700 1 -- 172.21.15.134:0/3926864797 wait complete. 2024-01-19T06:26:16.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.021+0000 7f57d62bd700 1 -- 172.21.15.134:0/1347927011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d00fd750 msgr2=0x7f57d00ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:16.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.021+0000 7f57d62bd700 1 --2- 172.21.15.134:0/1347927011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d00fd750 0x7f57d00ffb30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f57c0004940 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:16.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 -- 172.21.15.134:0/1347927011 shutdown_connections 2024-01-19T06:26:16.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 --2- 172.21.15.134:0/1347927011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d0100070 0x7f57d0102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 --2- 172.21.15.134:0/1347927011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d00fd750 0x7f57d00ffb30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 -- 172.21.15.134:0/1347927011 >> 172.21.15.134:0/1347927011 conn(0x7f57d00f7190 msgr2=0x7f57d00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 -- 172.21.15.134:0/1347927011 shutdown_connections 2024-01-19T06:26:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.022+0000 7f57d62bd700 1 -- 172.21.15.134:0/1347927011 wait complete. 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 Processor -- start 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 -- start start 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d010db80 0x7f57d010ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d010e040 con 0x7f57d0100070 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.023+0000 7f57d62bd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d010e180 con 0x7f57d010db80 2024-01-19T06:26:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40470/0 (socket says 172.21.15.134:40470) 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 -- 172.21.15.134:0/3060102590 learned_addr learned my addr 172.21.15.134:0/3060102590 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cf7fe700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d010db80 0x7f57d010ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 -- 172.21.15.134:0/3060102590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d010db80 msgr2=0x7f57d010ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d010db80 0x7f57d010ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 -- 172.21.15.134:0/3060102590 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0005040 con 0x7f57d0100070 2024-01-19T06:26:16.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.024+0000 7f57cffff700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f57c0000c00 tx=0x7f57c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:16.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.026+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c000e070 con 0x7f57d0100070 2024-01-19T06:26:16.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.026+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57c0004290 con 0x7f57d0100070 2024-01-19T06:26:16.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.026+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d010f020 con 0x7f57d0100070 2024-01-19T06:26:16.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.026+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c001cd90 con 0x7f57d0100070 2024-01-19T06:26:16.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.026+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d010f4d0 con 0x7f57d0100070 2024-01-19T06:26:16.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.028+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d01075d0 con 0x7f57d0100070 2024-01-19T06:26:16.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.029+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f57c001c5e0 con 0x7f57d0100070 2024-01-19T06:26:16.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.029+0000 7f57ccff9700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 0x7f57b806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:16.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.029+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f57c0088360 con 0x7f57d0100070 2024-01-19T06:26:16.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.029+0000 7f57cf7fe700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 0x7f57b806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:16.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.030+0000 7f57cf7fe700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 0x7f57b806b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009b80 tx=0x7f57c400b040 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:16.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.034+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f57c0058270 con 0x7f57d0100070 2024-01-19T06:26:16.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:16.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:16 smithi134 ceph-mon[28283]: from='client.? 172.21.15.134:0/3926864797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T06:26:16.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.383+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f57d01056a0 con 0x7f57b8068f70 2024-01-19T06:26:16.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.385+0000 7f57ccff9700 1 -- 172.21.15.134:0/3060102590 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f57d01056a0 con 0x7f57b8068f70 2024-01-19T06:26:16.385 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-01-19T06:26:16.385 INFO:teuthology.orchestra.run.smithi134.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-19T06:26:16.385 INFO:teuthology.orchestra.run.smithi134.stdout: "in_progress": true, 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout: "services_complete": [], 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout: "progress": "0/2 daemons upgraded", 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 image on host smithi150", 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout: "is_paused": false 2024-01-19T06:26:16.386 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.387+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 msgr2=0x7f57b806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.387+0000 7f57d62bd700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 0x7f57b806b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009b80 tx=0x7f57c400b040 comp rx=0 tx=0).stop 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.387+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 msgr2=0x7f57d010d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.387+0000 7f57d62bd700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f57c0000c00 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 shutdown_connections 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f57b8068f70 0x7f57b806b420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57d0100070 0x7f57d010d640 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 --2- 172.21.15.134:0/3060102590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d010db80 0x7f57d010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:16.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 >> 172.21.15.134:0/3060102590 conn(0x7f57d00f7190 msgr2=0x7f57d00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:16.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 shutdown_connections 2024-01-19T06:26:16.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:16.388+0000 7f57d62bd700 1 -- 172.21.15.134:0/3060102590 wait complete. 2024-01-19T06:26:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:16.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-01-19T06:26:16.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:16.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:16.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T06:26:16.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' 2024-01-19T06:26:16.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:16 smithi150 ceph-mon[35328]: from='client.? 172.21.15.134:0/3926864797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T06:26:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:17 smithi134 ceph-mon[28283]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-19T06:26:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:17 smithi134 ceph-mon[28283]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-19T06:26:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:17 smithi134 ceph-mon[28283]: Upgrade: Need to upgrade myself (mgr.smithi134.qrxokr) 2024-01-19T06:26:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:17 smithi134 ceph-mon[28283]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:17 smithi134 ceph-mon[28283]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi150 2024-01-19T06:26:17.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:17 smithi150 ceph-mon[35328]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-19T06:26:17.433 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:17 smithi150 ceph-mon[35328]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-19T06:26:17.434 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:17 smithi150 ceph-mon[35328]: Upgrade: Need to upgrade myself (mgr.smithi134.qrxokr) 2024-01-19T06:26:17.434 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:17 smithi150 ceph-mon[35328]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:17.434 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:17 smithi150 ceph-mon[35328]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi150 2024-01-19T06:26:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:18 smithi134 ceph-mon[28283]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:18.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:18 smithi134 ceph-mon[28283]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:26:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:18 smithi150 ceph-mon[35328]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T06:26:18.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:18 smithi150 ceph-mon[35328]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T06:26:20.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:20 smithi134 ceph-mon[28283]: pgmap v220: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s wr, 216 op/s 2024-01-19T06:26:20.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:20 smithi150 ceph-mon[35328]: pgmap v220: 65 pgs: 65 active+clean; 34 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s wr, 216 op/s 2024-01-19T06:26:20.858 INFO:tasks.workunit.client.0.smithi134.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: git switch -c 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr:Or undo this operation with: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: git switch - 2024-01-19T06:26:20.859 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.860 INFO:tasks.workunit.client.0.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-19T06:26:20.860 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:26:20.860 INFO:tasks.workunit.client.0.smithi134.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-19T06:26:20.863 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-19T06:26:20.895 INFO:tasks.workunit.client.0.smithi134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-19T06:26:20.897 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-19T06:26:20.897 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-19T06:26:21.676 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-19T06:26:21.737 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-19T06:26:21.777 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-19T06:26:21.779 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-19T06:26:21.779 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-19T06:26:21.818 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-19T06:26:21.821 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:26:21.821 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-19T06:26:21.877 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-01-19T06:26:21.878 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-19T06:26:21.878 DEBUG:teuthology.orchestra.run.smithi134:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-01-19T06:26:21.937 INFO:tasks.workunit.client.0.smithi134.stderr:+ mkdir -p fsstress 2024-01-19T06:26:21.939 INFO:tasks.workunit.client.0.smithi134.stderr:+ pushd fsstress 2024-01-19T06:26:21.939 INFO:tasks.workunit.client.0.smithi134.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-01-19T06:26:21.940 INFO:tasks.workunit.client.0.smithi134.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-19T06:26:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:22 smithi134 ceph-mon[28283]: pgmap v221: 65 pgs: 65 active+clean; 46 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s wr, 358 op/s 2024-01-19T06:26:22.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:22 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:22 smithi150 ceph-mon[35328]: pgmap v221: 65 pgs: 65 active+clean; 46 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s wr, 358 op/s 2024-01-19T06:26:22.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:22 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:22.487 INFO:tasks.workunit.client.0.smithi134.stderr:+ tar xzf ltp-full.tgz 2024-01-19T06:26:24.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:24 smithi134 ceph-mon[28283]: pgmap v222: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 437 op/s 2024-01-19T06:26:24.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:24 smithi150 ceph-mon[35328]: pgmap v222: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 437 op/s 2024-01-19T06:26:26.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:26 smithi134 ceph-mon[28283]: pgmap v223: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 448 op/s 2024-01-19T06:26:26.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:26 smithi150 ceph-mon[35328]: pgmap v223: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 448 op/s 2024-01-19T06:26:28.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:28 smithi134 ceph-mon[28283]: pgmap v224: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 448 op/s 2024-01-19T06:26:28.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:28 smithi150 ceph-mon[35328]: pgmap v224: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 448 op/s 2024-01-19T06:26:30.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:30 smithi134 ceph-mon[28283]: pgmap v225: 65 pgs: 65 active+clean; 77 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.7 MiB/s wr, 540 op/s 2024-01-19T06:26:30.435 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:30 smithi150 ceph-mon[35328]: pgmap v225: 65 pgs: 65 active+clean; 77 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.7 MiB/s wr, 540 op/s 2024-01-19T06:26:32.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:32 smithi134 ceph-mon[28283]: pgmap v226: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.7 MiB/s wr, 376 op/s 2024-01-19T06:26:32.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:32 smithi150 ceph-mon[35328]: pgmap v226: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.7 MiB/s wr, 376 op/s 2024-01-19T06:26:34.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:34 smithi134 ceph-mon[28283]: pgmap v227: 65 pgs: 65 active+clean; 94 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 270 op/s 2024-01-19T06:26:34.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:34 smithi150 ceph-mon[35328]: pgmap v227: 65 pgs: 65 active+clean; 94 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 4.2 MiB/s wr, 270 op/s 2024-01-19T06:26:36.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:36 smithi134 ceph-mon[28283]: pgmap v228: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s wr, 264 op/s 2024-01-19T06:26:36.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:36 smithi150 ceph-mon[35328]: pgmap v228: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.1 MiB/s wr, 264 op/s 2024-01-19T06:26:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:37 smithi134 ceph-mon[28283]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:37.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:37 smithi150 ceph-mon[35328]: from='mgr.14206 172.21.15.134:0/2220355568' entity='mgr.smithi134.qrxokr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T06:26:38.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:26:38 smithi134 ceph-mon[28283]: pgmap v229: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 386 op/s 2024-01-19T06:26:38.432 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Jan 19 06:26:38 smithi150 ceph-mon[35328]: pgmap v229: 65 pgs: 65 active+clean; 115 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.2 MiB/s wr, 386 op/s 2024-01-19T06:26:38.694 INFO:tasks.workunit.client.1.smithi150.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-01-19T06:26:38.694 INFO:tasks.workunit.client.1.smithi150.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-19T06:26:38.694 INFO:tasks.workunit.client.1.smithi150.stderr:+ make 2024-01-19T06:26:39.654 INFO:tasks.workunit.client.1.smithi150.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-01-19T06:26:46.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.508+0000 7f52d42f8700 1 -- 172.21.15.134:0/612997417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc066920 msgr2=0x7f52cc0ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:46.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.508+0000 7f52d42f8700 1 --2- 172.21.15.134:0/612997417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc066920 0x7f52cc0ff7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f52c8004940 tx=0x7f52c8009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.508+0000 7f52d42f8700 1 -- 172.21.15.134:0/612997417 shutdown_connections 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 --2- 172.21.15.134:0/612997417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52cc0ffd70 0x7f52cc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 --2- 172.21.15.134:0/612997417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc066920 0x7f52cc0ff7a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 -- 172.21.15.134:0/612997417 >> 172.21.15.134:0/612997417 conn(0x7f52cc0f71d0 msgr2=0x7f52cc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 -- 172.21.15.134:0/612997417 shutdown_connections 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 -- 172.21.15.134:0/612997417 wait complete. 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 Processor -- start 2024-01-19T06:26:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 -- start start 2024-01-19T06:26:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52cc066920 0x7f52cc10d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.509+0000 7f52d42f8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc10e180 con 0x7f52cc0ffd70 2024-01-19T06:26:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d42f8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc10e9c0 con 0x7f52cc066920 2024-01-19T06:26:46.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39638/0 (socket says 172.21.15.134:39638) 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 -- 172.21.15.134:0/1385292970 learned_addr learned my addr 172.21.15.134:0/1385292970 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 -- 172.21.15.134:0/1385292970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52cc066920 msgr2=0x7f52cc10d5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52cc066920 0x7f52cc10d5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 -- 172.21.15.134:0/1385292970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8005040 con 0x7f52cc0ffd70 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d1893700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00b760 tx=0x7f52bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52bc00f6f0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc10eca0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52bc00fd00 con 0x7f52cc0ffd70 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52bc011dd0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.510+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc10f1e0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.511+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc107660 con 0x7f52cc0ffd70 2024-01-19T06:26:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.512+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f52bc011600 con 0x7f52cc0ffd70 2024-01-19T06:26:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.512+0000 7f52c2ffd700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 0x7f52b806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.512+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f52bc086ab0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.512+0000 7f52d2094700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 0x7f52b806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:46.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.513+0000 7f52d2094700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 0x7f52b806b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52c8004890 tx=0x7f52c8009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:46.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.518+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f52bc0569c0 con 0x7f52cc0ffd70 2024-01-19T06:26:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.780+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52cc10f570 con 0x7f52b8068f60 2024-01-19T06:26:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.781+0000 7f52c2ffd700 1 -- 172.21.15.134:0/1385292970 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f52cc10f570 con 0x7f52b8068f60 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 msgr2=0x7f52b806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 0x7f52b806b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52c8004890 tx=0x7f52c8009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 msgr2=0x7f52cc10daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00b760 tx=0x7f52bc0091a0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 shutdown_connections 2024-01-19T06:26:46.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f52b8068f60 0x7f52b806b410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52cc066920 0x7f52cc10d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 --2- 172.21.15.134:0/1385292970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52cc0ffd70 0x7f52cc10daf0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 >> 172.21.15.134:0/1385292970 conn(0x7f52cc0f71d0 msgr2=0x7f52cc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 shutdown_connections 2024-01-19T06:26:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.784+0000 7f52d42f8700 1 -- 172.21.15.134:0/1385292970 wait complete. 2024-01-19T06:26:46.793 INFO:teuthology.orchestra.run.smithi134.stdout:true 2024-01-19T06:26:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 -- 172.21.15.134:0/860318315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fd0f0 msgr2=0x7f333c0fd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 --2- 172.21.15.134:0/860318315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fd0f0 0x7f333c0fd500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3330004940 tx=0x7f3330009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 -- 172.21.15.134:0/860318315 shutdown_connections 2024-01-19T06:26:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 --2- 172.21.15.134:0/860318315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f333c0fda40 0x7f333c0fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 --2- 172.21.15.134:0/860318315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fd0f0 0x7f333c0fd500 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 -- 172.21.15.134:0/860318315 >> 172.21.15.134:0/860318315 conn(0x7f333c0f69d0 msgr2=0x7f333c0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 -- 172.21.15.134:0/860318315 shutdown_connections 2024-01-19T06:26:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.888+0000 7f3342b62700 1 -- 172.21.15.134:0/860318315 wait complete. 2024-01-19T06:26:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 Processor -- start 2024-01-19T06:26:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 -- start start 2024-01-19T06:26:46.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f333c0fd0f0 0x7f333c1105b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0fbbd0 con 0x7f333c0fda40 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.889+0000 7f3342b62700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c0fbd10 con 0x7f333c0fd0f0 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39660/0 (socket says 172.21.15.134:39660) 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 -- 172.21.15.134:0/2184825341 learned_addr learned my addr 172.21.15.134:0/2184825341 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 -- 172.21.15.134:0/2184825341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f333c0fd0f0 msgr2=0x7f333c1105b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f333c0fd0f0 0x7f333c1105b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 -- 172.21.15.134:0/2184825341 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330005040 con 0x7f333c0fda40 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f333bfff700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f332800b760 tx=0x7f33280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f332800f6f0 con 0x7f333c0fda40 2024-01-19T06:26:46.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f332800fd00 con 0x7f333c0fda40 2024-01-19T06:26:46.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c1131e0 con 0x7f333c0fda40 2024-01-19T06:26:46.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3328011dd0 con 0x7f333c0fda40 2024-01-19T06:26:46.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.890+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c113720 con 0x7f333c0fda40 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.892+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333c106590 con 0x7f333c0fda40 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.892+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f3328011600 con 0x7f333c0fda40 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.892+0000 7f33397fa700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 0x7f332c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.892+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3328086ab0 con 0x7f333c0fda40 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.893+0000 7f33408fe700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 0x7f332c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:46.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.893+0000 7f33408fe700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 0x7f332c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3330004890 tx=0x7f3330009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:46.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:46.898+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3328052f30 con 0x7f333c0fda40 2024-01-19T06:26:47.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.131+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f333c1046c0 con 0x7f332c068f60 2024-01-19T06:26:47.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.132+0000 7f33397fa700 1 -- 172.21.15.134:0/2184825341 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f333c1046c0 con 0x7f332c068f60 2024-01-19T06:26:47.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 msgr2=0x7f332c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 0x7f332c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3330004890 tx=0x7f3330009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 msgr2=0x7f333c112b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f332800b760 tx=0x7f33280091a0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 shutdown_connections 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f332c068f60 0x7f332c06b410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f333c0fd0f0 0x7f333c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 --2- 172.21.15.134:0/2184825341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f333c0fda40 0x7f333c112b00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 >> 172.21.15.134:0/2184825341 conn(0x7f333c0f69d0 msgr2=0x7f333c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 shutdown_connections 2024-01-19T06:26:47.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.134+0000 7f3342b62700 1 -- 172.21.15.134:0/2184825341 wait complete. 2024-01-19T06:26:47.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 -- 172.21.15.134:0/104107497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fcd00 msgr2=0x7f9d680fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 --2- 172.21.15.134:0/104107497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fcd00 0x7f9d680fd150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9d54004940 tx=0x7f9d54009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 -- 172.21.15.134:0/104107497 shutdown_connections 2024-01-19T06:26:47.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 --2- 172.21.15.134:0/104107497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fcd00 0x7f9d680fd150 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 --2- 172.21.15.134:0/104107497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680fbaf0 0x7f9d680fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 -- 172.21.15.134:0/104107497 >> 172.21.15.134:0/104107497 conn(0x7f9d680f7190 msgr2=0x7f9d680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 -- 172.21.15.134:0/104107497 shutdown_connections 2024-01-19T06:26:47.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.236+0000 7f9d6f257700 1 -- 172.21.15.134:0/104107497 wait complete. 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 Processor -- start 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 -- start start 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680fcd00 0x7f9d68112870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680661a0 con 0x7f9d680fbaf0 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6f257700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680662e0 con 0x7f9d680fcd00 2024-01-19T06:26:47.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6cff3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6cff3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39672/0 (socket says 172.21.15.134:39672) 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6cff3700 1 -- 172.21.15.134:0/81214524 learned_addr learned my addr 172.21.15.134:0/81214524 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6cff3700 1 -- 172.21.15.134:0/81214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680fcd00 msgr2=0x7f9d68112870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.237+0000 7f9d6cff3700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680fcd00 0x7f9d68112870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d6cff3700 1 -- 172.21.15.134:0/81214524 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d54005040 con 0x7f9d680fbaf0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d6cff3700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b760 tx=0x7f9d5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c00f6f0 con 0x7f9d680fbaf0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d68112f00 con 0x7f9d680fbaf0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d5c00fd00 con 0x7f9d680fbaf0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d5c011d70 con 0x7f9d680fbaf0 2024-01-19T06:26:47.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.238+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d681133b0 con 0x7f9d680fbaf0 2024-01-19T06:26:47.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.239+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6804bb80 con 0x7f9d680fbaf0 2024-01-19T06:26:47.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.240+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f9d5c00f850 con 0x7f9d680fbaf0 2024-01-19T06:26:47.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.240+0000 7f9d657fa700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 0x7f9d5806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.240+0000 7f9d67fff700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 0x7f9d5806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.240+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9d5c086a40 con 0x7f9d680fbaf0 2024-01-19T06:26:47.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.241+0000 7f9d67fff700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 0x7f9d5806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d54004890 tx=0x7f9d54009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.245+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9d5c052ec0 con 0x7f9d680fbaf0 2024-01-19T06:26:47.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.452+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d68101650 con 0x7f9d58068f60 2024-01-19T06:26:47.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.462+0000 7f9d657fa700 1 -- 172.21.15.134:0/81214524 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f9d68101650 con 0x7f9d58068f60 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:alertmanager.smithi134 smithi134 *:9093,9094 running (5m) 91s ago 7m 24.4M - 0.25.0 c8568f914cd2 09f464d8c526 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi134 smithi134 running (7m) 91s ago 7m 6665k - 17.2.7-268-g6441305d c3a232f785d8 434084e15158 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:crash.smithi150 smithi150 running (6m) 98s ago 6m 6660k - 17.2.7-268-g6441305d c3a232f785d8 1aee22edee9f 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:grafana.smithi134 smithi134 *:3000 running (5m) 91s ago 6m 84.4M - 9.4.7 954c08fa6188 d3ec6951f4ff 2024-01-19T06:26:47.462 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi134.chvejw smithi134 running (110s) 91s ago 110s 11.3M - 17.2.7-268-g6441305d c3a232f785d8 86973986f1f1 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi134.higalz smithi134 running (118s) 91s ago 117s 10.4M - 17.2.7-268-g6441305d c3a232f785d8 86c9152ca3d8 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi150.tsxxhg smithi150 running (108s) 98s ago 107s 13.5M - 17.2.7-268-g6441305d c3a232f785d8 bc505d3ee14e 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mds.cephfs.smithi150.xqfgpx smithi150 running (115s) 98s ago 115s 13.3M - 17.2.7-268-g6441305d c3a232f785d8 0c977cab8f03 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi134.qrxokr smithi134 *:9283 running (8m) 91s ago 8m 481M - 17.2.7-268-g6441305d c3a232f785d8 6f69c250d5a7 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.smithi150.xkxrco smithi150 *:8443,9283 running (6m) 98s ago 6m 433M - 17.2.7-268-g6441305d c3a232f785d8 d6d4dd5f44c3 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi134 smithi134 running (8m) 91s ago 8m 45.0M 2048M 17.2.7-268-g6441305d c3a232f785d8 328c6cb50ef7 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:mon.smithi150 smithi150 running (6m) 98s ago 6m 35.6M 2048M 17.2.7-268-g6441305d c3a232f785d8 36884ce908a9 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi134 smithi134 *:9100 running (7m) 91s ago 7m 15.7M - 1.5.0 0da6a335fe13 9ff3509f6e9d 2024-01-19T06:26:47.463 INFO:teuthology.orchestra.run.smithi134.stdout:node-exporter.smithi150 smithi150 *:9100 running (6m) 98s ago 6m 15.6M - 1.5.0 0da6a335fe13 f5edbad1d4e4 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (5m) 91s ago 5m 49.0M 4288M 17.2.7-268-g6441305d c3a232f785d8 c42a610a3422 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi134 running (5m) 91s ago 5m 47.5M 4288M 17.2.7-268-g6441305d c3a232f785d8 980268e5252b 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi134 running (4m) 91s ago 4m 44.5M 4288M 17.2.7-268-g6441305d c3a232f785d8 cbc144916de2 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi150 running (4m) 98s ago 4m 45.2M 5312M 17.2.7-268-g6441305d c3a232f785d8 799be7524f32 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 smithi150 running (3m) 98s ago 3m 44.2M 5312M 17.2.7-268-g6441305d c3a232f785d8 4e0d71b857f1 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 smithi150 running (3m) 98s ago 3m 40.6M 5312M 17.2.7-268-g6441305d c3a232f785d8 175a03fc551b 2024-01-19T06:26:47.464 INFO:teuthology.orchestra.run.smithi134.stdout:prometheus.smithi134 smithi134 *:9095 running (5m) 91s ago 6m 56.4M - 2.43.0 a07b618ecd1d e5681937937a 2024-01-19T06:26:47.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 msgr2=0x7f9d5806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 0x7f9d5806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d54004890 tx=0x7f9d54009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 msgr2=0x7f9d68110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00b760 tx=0x7f9d5c0091a0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 shutdown_connections 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f9d58068f60 0x7f9d5806b410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d680fbaf0 0x7f9d68110320 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 --2- 172.21.15.134:0/81214524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d680fcd00 0x7f9d68112870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 >> 172.21.15.134:0/81214524 conn(0x7f9d680f7190 msgr2=0x7f9d680fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 shutdown_connections 2024-01-19T06:26:47.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.465+0000 7f9d6f257700 1 -- 172.21.15.134:0/81214524 wait complete. 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 -- 172.21.15.134:0/1352546128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00780ffd90 msgr2=0x7f00781021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 --2- 172.21.15.134:0/1352546128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00780ffd90 0x7f00781021d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0068004940 tx=0x7f0068009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 -- 172.21.15.134:0/1352546128 shutdown_connections 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 --2- 172.21.15.134:0/1352546128 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00780ffd90 0x7f00781021d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 --2- 172.21.15.134:0/1352546128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0078066900 0x7f00780ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 -- 172.21.15.134:0/1352546128 >> 172.21.15.134:0/1352546128 conn(0x7f00780f71b0 msgr2=0x7f00780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.566+0000 7f007f93a700 1 -- 172.21.15.134:0/1352546128 shutdown_connections 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 -- 172.21.15.134:0/1352546128 wait complete. 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 Processor -- start 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 -- start start 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00780ffd90 0x7f007810daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007810e180 con 0x7f0078066900 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007f93a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007810e9c0 con 0x7f00780ffd90 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007d6d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007d6d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39688/0 (socket says 172.21.15.134:39688) 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.567+0000 7f007d6d6700 1 -- 172.21.15.134:0/542476157 learned_addr learned my addr 172.21.15.134:0/542476157 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007d6d6700 1 -- 172.21.15.134:0/542476157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00780ffd90 msgr2=0x7f007810daf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007d6d6700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00780ffd90 0x7f007810daf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007d6d6700 1 -- 172.21.15.134:0/542476157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0068005040 con 0x7f0078066900 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007d6d6700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f007400b580 tx=0x7f00740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f007400f6f0 con 0x7f0078066900 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007810eca0 con 0x7f0078066900 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f007400fd00 con 0x7f0078066900 2024-01-19T06:26:47.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0074011e30 con 0x7f0078066900 2024-01-19T06:26:47.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.568+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007810f1e0 con 0x7f0078066900 2024-01-19T06:26:47.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.569+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078107660 con 0x7f0078066900 2024-01-19T06:26:47.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.570+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f007400f850 con 0x7f0078066900 2024-01-19T06:26:47.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.570+0000 7f006dffb700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 0x7f006406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.570+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0074086b00 con 0x7f0078066900 2024-01-19T06:26:47.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.570+0000 7f007ced5700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 0x7f006406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.571+0000 7f007ced5700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 0x7f006406b420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f006800b3b0 tx=0x7f0068009e30 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.574+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0074056a10 con 0x7f0078066900 2024-01-19T06:26:47.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.845+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f007810f490 con 0x7f0078066900 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.846+0000 7f006dffb700 1 -- 172.21.15.134:0/542476157 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f00740565a0 con 0x7f0078066900 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: "mon": { 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": { 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:47.845 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": { 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 6 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: "mds": { 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 4 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: "overall": { 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 14 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-01-19T06:26:47.846 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-01-19T06:26:47.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 msgr2=0x7f006406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 0x7f006406b420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f006800b3b0 tx=0x7f0068009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 msgr2=0x7f007810d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f007400b580 tx=0x7f00740091a0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 shutdown_connections 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f0064068f70 0x7f006406b420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0078066900 0x7f007810d5b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 --2- 172.21.15.134:0/542476157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00780ffd90 0x7f007810daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 >> 172.21.15.134:0/542476157 conn(0x7f00780f71b0 msgr2=0x7f00780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 shutdown_connections 2024-01-19T06:26:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.848+0000 7f007f93a700 1 -- 172.21.15.134:0/542476157 wait complete. 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3966763825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 msgr2=0x7f3d880fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3966763825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d880fbf00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3d78004940 tx=0x7f3d78009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3966763825 shutdown_connections 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3966763825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d880fcd00 0x7f3d880fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3966763825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d880fbf00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.952+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3966763825 >> 172.21.15.134:0/3966763825 conn(0x7f3d880f71d0 msgr2=0x7f3d880f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3966763825 shutdown_connections 2024-01-19T06:26:47.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3966763825 wait complete. 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 Processor -- start 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 -- start start 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d880fcd00 0x7f3d88114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88066890 con 0x7f3d880fbaf0 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.953+0000 7f3d8d8d1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880669d0 con 0x7f3d880fcd00 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39708/0 (socket says 172.21.15.134:39708) 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 -- 172.21.15.134:0/3264756942 learned_addr learned my addr 172.21.15.134:0/3264756942 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 -- 172.21.15.134:0/3264756942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d880fcd00 msgr2=0x7f3d88114c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d880fcd00 0x7f3d88114c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 -- 172.21.15.134:0/3264756942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78005040 con 0x7f3d880fbaf0 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d86ffd700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d78000c00 tx=0x7f3d78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d7800e030 con 0x7f3d880fbaf0 2024-01-19T06:26:47.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d78004290 con 0x7f3d880fbaf0 2024-01-19T06:26:47.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d7801c400 con 0x7f3d880fbaf0 2024-01-19T06:26:47.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d881151c0 con 0x7f3d880fbaf0 2024-01-19T06:26:47.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.954+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d88115620 con 0x7f3d880fbaf0 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.955+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8804bb80 con 0x7f3d880fbaf0 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.956+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 86449+0+0 (secure 0 0 0) 0x7f3d78013420 con 0x7f3d880fbaf0 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.956+0000 7f3d6f7fe700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 0x7f3d7006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.956+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3d780885e0 con 0x7f3d880fbaf0 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.956+0000 7f3d867fc700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 0x7f3d7006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T06:26:47.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.957+0000 7f3d867fc700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 0x7f3d7006b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009740 tx=0x7f3d7c006c60 comp rx=0 tx=0).ready entity=mgr.14206 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T06:26:47.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:47.962+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3d78054b00 con 0x7f3d880fbaf0 2024-01-19T06:26:48.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.193+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 --> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d88101650 con 0x7f3d70068f60 2024-01-19T06:26:48.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.194+0000 7f3d6f7fe700 1 -- 172.21.15.134:0/3264756942 <== mgr.14206 v2:172.21.15.134:6800/233880040 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f3d88101650 con 0x7f3d70068f60 2024-01-19T06:26:48.194 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "in_progress": true, 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "services_complete": [], 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "progress": "0/2 daemons upgraded", 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 image on host smithi150", 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout: "is_paused": false 2024-01-19T06:26:48.195 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 msgr2=0x7f3d7006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 0x7f3d7006b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009740 tx=0x7f3d7c006c60 comp rx=0 tx=0).stop 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 msgr2=0x7f3d88112730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d78000c00 tx=0x7f3d78009e30 comp rx=0 tx=0).stop 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 shutdown_connections 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:6800/233880040,v1:172.21.15.134:6801/233880040] conn(0x7f3d70068f60 0x7f3d7006b410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d880fbaf0 0x7f3d88112730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:48.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 --2- 172.21.15.134:0/3264756942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d880fcd00 0x7f3d88114c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T06:26:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 >> 172.21.15.134:0/3264756942 conn(0x7f3d880f71d0 msgr2=0x7f3d880fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T06:26:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.196+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 shutdown_connections 2024-01-19T06:26:48.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:26:48.197+0000 7f3d8d8d1700 1 -- 172.21.15.134:0/3264756942 wait complete. 2024-01-19T06:27:01.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:01.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:01.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:01.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:01.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:01.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:02.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:02.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:02.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:02.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:02.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:02.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:02.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:02.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:03.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:03.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:03.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:03.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:03.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:03.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:03.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:04.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:04.438+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:04.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:04.438+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:04.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:04.438+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:04.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:04.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:04.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:04.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:04.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:04.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:04.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:05.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:05.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:05.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:05.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:05.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:05.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:05.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:05.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:05.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:05.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:05.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:05.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:06.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:06.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:06.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:06.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:06.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:06.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:06.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:06.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:06.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:06.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:06.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:06.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:07.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:07.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:07.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:07.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:07.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:07.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:07.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:07.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:07.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:07.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:07.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:07.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:07.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:08.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:08.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:08.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:08.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:08.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:08.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:08.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:08.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:08.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:08.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:08.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:08.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:09.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:09.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:09.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:09.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:09.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:09.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:09.432+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:09.432+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:09.432+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:10.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:10.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:10.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:10.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:10.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:10.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:10.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:10.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:10.442+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:10.442+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:10.442+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:11.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:11.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:11.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:11.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:11.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:11.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:11.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:11.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:11.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:11.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:11.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:12.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:12.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:12.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:12.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:12.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:12.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:12.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:12.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:12.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:12.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:12.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:12.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:12.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:12.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:12.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:13.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:13.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:13.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:13.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:13.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:13.408+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:13.408+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:13.408+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:13.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:13.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:13.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:14.524 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:14.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:14.525 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:14.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:14.525 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:14.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:14.525 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:14.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:14.525 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:14.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:14.525 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:14.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:14.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:14.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:14.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:14.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:14.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:14.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:15.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:15.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:15.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:15.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:15.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:15.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:15.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:15.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:15.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:15.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:15.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:15.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:16.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:16.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:16.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:16.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:16.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:16.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:16.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:16.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:16.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:16.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:16.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:16.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:16.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:16.548+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:16.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:16.548+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:16.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:16.548+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:17.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:17.521+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:17.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:17.521+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:17.521+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:17.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:17.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:17.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:17.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:17.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:17.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:18.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:18.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:18.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:18.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:18.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:18.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:18.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:18.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:18.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:18.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:18.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:18.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:18.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:18.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:18.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:18.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:18.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:18.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:19.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:19.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:19.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:19.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:19.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:19.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:19.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:19.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:19.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:19.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:19.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:19.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:19.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:20.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:20.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:20.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:20.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:20.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:20.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:20.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:20.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:21.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:21.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:21.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:21.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:21.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:21.347+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:21.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:21.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:21.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:21.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:21.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:21.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:22.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:22.338+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:22.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:22.338+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:22.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:22.338+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:22.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:22.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:22.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:22.527+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:22.527+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:22.527+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:23.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:23.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:23.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:23.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:23.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:23.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:23.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:23.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:23.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:23.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:23.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:23.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:24.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:24.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:24.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:24.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:24.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:24.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:24.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:24.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:24.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:24.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:25.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:25.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:25.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:25.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:25.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:25.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:25.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:25.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:25.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:25.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:25.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:26.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:26.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:26.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:26.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:26.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:26.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:26.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:26.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:26.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:26.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:26.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:27.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:27.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:27.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:27.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:27.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:27.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:27.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:27.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:27.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:27.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:27.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:27.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:27.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:28.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:28.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:28.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:28.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:28.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:28.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:28.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:28.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:28.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:28.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:28.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:28.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:29.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:29.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:29.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:29.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:29.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:29.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:29.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:29.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:29.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:29.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:29.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:29.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:29.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:29.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:29.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:29.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:29.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:29.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:30.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:30.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:30.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:30.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:30.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:30.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:30.447 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:30.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:30.447 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:30.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:30.448 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:30.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:30.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:30.497+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:30.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:30.497+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:30.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:30.497+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:31.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:31.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:31.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:31.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:31.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:31.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:31.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:31.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:31.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:31.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:31.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:31.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:31.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:32.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:32.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:32.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:32.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:32.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:32.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:32.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:32.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:32.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:32.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:32.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:32.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:33.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:33.478+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:33.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:33.478+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:33.478+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:33.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:33.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:33.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:33.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:33.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:33.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:34.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:34.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:34.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:34.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:34.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:34.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:34.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:34.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:34.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:34.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:34.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:34.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:35.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:35.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:35.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:35.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:35.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:35.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:35.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:35.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:35.369+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:35.369+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:35.369+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:36.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:36.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:36.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:36.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:36.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:36.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:36.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:36.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:36.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:36.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:36.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:36.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:37.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:37.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:37.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:37.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:37.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:37.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:37.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:37.445+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:37.445+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:37.445+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:38.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:38.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:38.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:38.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:38.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:38.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:38.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:38.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:38.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:38.471+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:38.471+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:38.471+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:39.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:39.325+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:39.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:39.325+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:39.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:39.325+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:39.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:39.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:39.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:39.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:39.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:39.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:40.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:40.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:40.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:40.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:40.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:40.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:40.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:40.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:40.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:40.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:41.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:41.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:41.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:41.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:41.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:41.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:41.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:41.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:41.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:41.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:41.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:41.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:41.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:42.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:42.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:42.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:42.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:42.467+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:42.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:42.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:42.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:42.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:42.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:42.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:43.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:43.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:43.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:43.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:43.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:43.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:43.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:43.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:43.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:43.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:43.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:43.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:43.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:44.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:44.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:44.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:44.488+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:44.488+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:44.488+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:44.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:44.485+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:44.485+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:44.485+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:45.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:45.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:45.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:45.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:45.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:45.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:45.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:45.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:45.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:45.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:46.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:46.504+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:46.504+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:46.504+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:46.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:46.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:46.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:46.429+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:46.429+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:46.429+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:47.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:47.515+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:47.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:47.515+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:47.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:47.515+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:47.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:47.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:47.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:47.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:47.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:47.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:47.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:48.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:48.562+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:48.562+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:48.562+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:48.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:48.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:48.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:48.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:48.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:48.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:49.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:49.540+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:49.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:49.540+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:49.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:49.540+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:49.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:49.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:49.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:49.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:49.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:49.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:50.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:50.552+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:50.552+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:50.552+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:50.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:50.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:50.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:50.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:50.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:50.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:50.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:51.523 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:51.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:51.523 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:51.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:51.523 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:51.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:51.524 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:51.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:51.524 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:51.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:51.524 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:51.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:51.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:51.523+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:51.523+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:51.523+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:52.420 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:52.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:52.420 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:52.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:52.420 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:52.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:52.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:52.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:52.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:52.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:52.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:52.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:52.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:52.420+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:52.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:52.420+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:52.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:52.420+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:53.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:53.492+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:53.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:53.492+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:53.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:53.492+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:53.512 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:53.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:53.512 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:53.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:53.513 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:53.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:53.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:53.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:53.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:54.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:54.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:54.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:54.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:54.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:54.478 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:54.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:54.477+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:54.477+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:54.477+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:55.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:55.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:55.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:55.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:55.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:55.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:55.501 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:55.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:55.501 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:55.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:55.502 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:55.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:55.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:55.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:55.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:55.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:55.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:55.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:56.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:56.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:56.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:56.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:56.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:56.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:56.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:56.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:56.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:56.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:56.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:56.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:57.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:57.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:57.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:57.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:57.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:57.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:57.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:57.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:57.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:57.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:57.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:57.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:57.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:57.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:57.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:57.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:57.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:57.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:58.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:58.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:58.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:58.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:58.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:58.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:58.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:58.468+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:58.939 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:58.468+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:58.939 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:58.468+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:58.939 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:58.505+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:58.939 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:58.505+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:58.939 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:58.505+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:59.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:59.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:59.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:59.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:59.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:27:59.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:27:59.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:59.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:59.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:59.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:59.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:27:59.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:27:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:59.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:59.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:27:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:27:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:27:59.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:00.513 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:00.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:00.513 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:00.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:00.513 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:00.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:00.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:00.512+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:00.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:00.512+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:00.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:00.512+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:00.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:00.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:00.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:01.493 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:01.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:01.493 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:01.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:01.493 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:01.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:01.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:01.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:01.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:01.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:01.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:01.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:01.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:01.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:01.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:02.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:02.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:02.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:02.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:02.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:02.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:02.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:02.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:02.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:02.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:02.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:02.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:02.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:02.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:02.524+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:03.480 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:03.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:03.481 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:03.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:03.481 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:03.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:03.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:03.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:03.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:03.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:03.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:03.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:03.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:03.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:03.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:04.522 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:04.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:04.522 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:04.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:04.522 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:04.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:04.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:04.521+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:04.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:04.521+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:04.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:04.521+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:04.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:04.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:04.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:05.517 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:05.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:05.517 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:05.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:05.517 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:05.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:05.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:05.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:05.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:05.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:05.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:05.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:05.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:05.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:05.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:05.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:05.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:05.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:06.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:06.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:06.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:06.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:06.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:06.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:06.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:06.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:06.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:06.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:06.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:06.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:06.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:06.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:06.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:06.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:06.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:06.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:07.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:07.133+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:07.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:07.133+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:07.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:07.133+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:07.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:07.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:07.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:07.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:07.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:07.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:07.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:07.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:07.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:08.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:08.085+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:08.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:08.085+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:08.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:08.085+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:08.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:08.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:08.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:08.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:08.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:08.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:08.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:08.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:08.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:08.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:08.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:08.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:09.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:09.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:09.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:09.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:09.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:09.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:09.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:09.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:09.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:09.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:10.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:10.053+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:10.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:10.053+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:10.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:10.053+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:10.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:10.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:10.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:10.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:10.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:10.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:10.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:10.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:10.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:10.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:10.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:10.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:11.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:11.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:11.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:11.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:11.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:11.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:11.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:11.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:11.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:11.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:11.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:11.596+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:11.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:11.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:11.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:11.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:11.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:11.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:12.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:12.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:12.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:12.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:12.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:12.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:12.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:12.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:12.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:12.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:12.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:12.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:12.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:12.513+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:13.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:13.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:13.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:13.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:13.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:13.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:13.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:13.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:13.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:13.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:13.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:13.500+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:13.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:13.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:13.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:14.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:14.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:14.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:14.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:14.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:14.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:14.812 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:14.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:14.813 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:14.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:14.813 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:14.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:14.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:14.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:14.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:14.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:14.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:14.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:15.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.014+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:15.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.014+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:15.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.014+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:15.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:15.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:15.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:15.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:15.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:15.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:15.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:15.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:15.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:16.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:16.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:16.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:15.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:16.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:16.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:16.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:16.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:16.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:16.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:16.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:16.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:16.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:17.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:16.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:17.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:16.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:17.424 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:16.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:17.911 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:17.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:17.912 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:17.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:17.912 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:17.501+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:17.912 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:17.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:17.912 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:17.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:17.912 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:17.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:18.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:17.911+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:18.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:17.911+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:18.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:17.911+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:18.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:18.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:18.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:18.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:18.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:18.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:18.863 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:18.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:18.864 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:18.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:18.864 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:18.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:19.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:18.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:19.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:18.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:19.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:18.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:19.899 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:19.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:19.899 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:19.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:19.900 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:19.505+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:19.900 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:19.727+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:19.900 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:19.727+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:19.900 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:19.727+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:20.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:19.899+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:20.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:19.899+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:20.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:19.899+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:20.874 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:20.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:20.874 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:20.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:20.874 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:20.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:20.875 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:20.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:20.875 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:20.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:20.875 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:20.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:21.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:20.873+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:21.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:20.873+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:21.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:20.873+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:21.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:21.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:21.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:21.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:21.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:21.856 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:21.538+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:22.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:21.855+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:22.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:21.855+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:22.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:21.855+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:22.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:22.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:22.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:22.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:22.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:22.858 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:22.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:23.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:22.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:23.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:22.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:23.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:22.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:23.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:23.461+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:23.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:23.461+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:23.863 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:23.461+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:23.864 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:23.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:23.864 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:23.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:23.864 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:23.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:24.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:23.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:24.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:23.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:24.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:23.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:24.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:24.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:24.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:24.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:24.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:24.838 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:24.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:25.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:24.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:25.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:24.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:25.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:24.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:25.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:25.433+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:25.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:25.433+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:25.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:25.433+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:26.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:25.836+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:26.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:25.836+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:26.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:25.836+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:25.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:25.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:25.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:26.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:26.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:26.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:26.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:26.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:26.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:26.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:27.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:26.801+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:27.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:26.801+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:27.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:26.801+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:27.778 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:27.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:27.778 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:27.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:27.778 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:27.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:27.779 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:27.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:27.779 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:27.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:27.779 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:27.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:28.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:27.778+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:28.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:27.778+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:28.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:27.778+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:28.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:28.542+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:28.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:28.542+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:28.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:28.542+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:28.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:28.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:28.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:28.765+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:28.765+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:28.765+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:29.575+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:29.575+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:29.575+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:29.826 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:29.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:29.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:29.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:29.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:29.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:29.827 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:29.760+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:29.827 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:29.760+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:29.827 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:29.760+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:30.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:30.550+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:30.550+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:30.550+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:30.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:30.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:30.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:30.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:30.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:30.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:30.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:31.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:31.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:31.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:31.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:31.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:31.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:31.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:32.038 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:31.768+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:32.038 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:31.768+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:32.038 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:31.768+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:32.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:32.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:32.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:32.519+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:32.519+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:32.798 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:32.519+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:33.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:32.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:33.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:32.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:33.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:32.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:33.771 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:33.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:33.771 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:33.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:33.771 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:33.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:33.771 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:33.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:33.772 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:33.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:33.772 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:33.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:34.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:33.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:33.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:33.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:34.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:34.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:34.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:34.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:34.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:34.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:34.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:34.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:34.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:34.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:34.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:34.508+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:35.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:35.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:35.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:35.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:35.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:35.785 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:35.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:36.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:35.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:35.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:35.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:36.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:36.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:36.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:36.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:36.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:36.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:36.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:36.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:36.802+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:36.802+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:36.802+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:37.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:37.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:37.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:37.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:37.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:37.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:37.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:38.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:37.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:38.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:37.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:38.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:37.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:38.772 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:38.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:38.772 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:38.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:38.772 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:38.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:38.772 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:38.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:38.773 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:38.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:38.773 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:38.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:39.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:38.771+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:39.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:38.771+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:39.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:38.771+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:39.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:39.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:39.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:39.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:39.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:39.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:39.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:39.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:39.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:39.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:40.772+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:40.772+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:40.772+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:40.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:40.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:40.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:40.522+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:40.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:40.522+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:40.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:40.522+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:41.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:41.757+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:41.757+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:41.757+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:41.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:41.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:41.626+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:41.528+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:41.528+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:41.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:41.528+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:28:42.029+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:28:42.803 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:42.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:42.804 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:42.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:42.804 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:42.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:42.804 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:42.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:42.804 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:42.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:42.804 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:42.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:43.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:42.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:42.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:42.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:43.834 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:43.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:43.834 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:43.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:43.834 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:43.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:43.835 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:43.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:43.835 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:43.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:43.835 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:43.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:44.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:43.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:43.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:43.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:44.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:44.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:44.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:44.587+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:44.587+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:44.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:44.587+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:45.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:44.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:45.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:44.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:45.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:44.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:45.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:45.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:45.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:45.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:45.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:45.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:45.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:46.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:45.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:46.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:45.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:45.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:46.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:46.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:46.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:46.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:46.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:46.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:47.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:28:47.030+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:28:47.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:46.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:47.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:46.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:47.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:46.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:47.844 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:47.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:47.845 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:47.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:47.845 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:47.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:47.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:47.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:47.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:47.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:47.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:47.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:48.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:47.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:48.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:47.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:48.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:47.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:48.889 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:48.482+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:48.889 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:48.482+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:48.890 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:48.482+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:48.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:48.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:48.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:48.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:48.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:48.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:49.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:48.889+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:48.889+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:48.889+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:49.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:49.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:49.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:49.484+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:49.484+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:49.861 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:49.484+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:50.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:49.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:50.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:49.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:50.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:49.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:50.812 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:50.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:50.812 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:50.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:50.812 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:50.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:50.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:50.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:50.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:50.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:50.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:50.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:51.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:50.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:51.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:50.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:51.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:50.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:51.804 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:51.509+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:51.804 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:51.509+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:51.804 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:51.509+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:51.804 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:51.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:51.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:51.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:51.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:51.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:51.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:51.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:51.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:51.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:51.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:51.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:52.152 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:28:52.030+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:52.524+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:52.524+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:52.524+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:52.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:52.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:52.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:53.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:52.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:53.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:52.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:52.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:53.824 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:53.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:53.824 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:53.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:53.825 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:53.541+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:53.825 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:53.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:53.825 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:53.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:53.825 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:53.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:54.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:53.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:53.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:53.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:54.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:54.506+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:54.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:54.506+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:54.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:54.506+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:54.830 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:54.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:54.830 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:54.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:54.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:54.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:55.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:54.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:55.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:54.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:55.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:54.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:55.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:55.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:55.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:55.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:55.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:55.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:55.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:56.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:55.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:56.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:55.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:56.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:55.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:56.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:56.784 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:56.784 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:56.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:56.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:56.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:56.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:56.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:56.491+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:57.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:28:57.031+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:28:57.043 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:56.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:57.043 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:56.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:57.043 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:56.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:57.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:57.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:57.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:57.666+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:57.666+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:57.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:57.666+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:57.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:57.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:57.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:57.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:57.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:57.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:58.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:58.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:58.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:58.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:58.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:58.780 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:58.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:59.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:58.779+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:59.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:58.779+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:59.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:58.779+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:59.518+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:59.518+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:28:59.518+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:59.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:59.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:28:59.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:28:59.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:00.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:59.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:00.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:59.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:00.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:28:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:28:59.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:00.815 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:00.602+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:00.816 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:00.602+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:00.816 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:00.602+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:00.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:00.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:00.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:00.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:00.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:00.490+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:01.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:00.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:01.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:00.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:01.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:00.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:01.846 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:01.532+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:01.846 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:01.532+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:01.846 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:01.532+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:01.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:01.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:01.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:01.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:01.847 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:01.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:02.153 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:02.031+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 56 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:02.153 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:01.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:02.154 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:01.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:02.154 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:01.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:02.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:02.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:02.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:02.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:02.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:02.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:02.842 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:02.529+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:02.842 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:02.529+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:02.842 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:02.529+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:03.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:02.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:03.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:02.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:03.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:02.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:03.811 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:03.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:03.811 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:03.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:03.811 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:03.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:03.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:03.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:03.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:03.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:03.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:03.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:04.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:03.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:04.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:03.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:04.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:03.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:04.852 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:04.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:04.853 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:04.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:04.853 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:04.450+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:04.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:04.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:04.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:04.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:04.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:04.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:05.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:04.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:05.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:04.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:05.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:04.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:05.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:05.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:05.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:05.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:05.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:05.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:05.894 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:05.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:05.894 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:05.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:05.894 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:05.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:06.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:05.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:06.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:05.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:06.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:05.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:06.892 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:06.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:06.892 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:06.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:06.892 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:06.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:06.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:06.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:06.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:06.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:06.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:06.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:07.156 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:07.031+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 85 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:07.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:06.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:07.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:06.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:07.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:06.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:07.486+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:07.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:07.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:07.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:07.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:08.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:07.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:08.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:07.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:08.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:07.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:08.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:08.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:08.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:08.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:08.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:08.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:08.850 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:08.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:08.850 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:08.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:08.850 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:08.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:09.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:09.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:09.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:09.494+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:09.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:09.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:09.858 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:09.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:10.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:09.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:10.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:09.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:10.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:09.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:10.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:10.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:10.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:10.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:10.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:10.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:10.839 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:10.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:10.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:10.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:10.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:10.533+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:11.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:10.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:11.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:10.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:11.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:10.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:11.793 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:11.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:11.794 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:11.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:11.794 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:11.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:11.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:11.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:11.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:12.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:11.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:12.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:11.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:12.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:11.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:12.055 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:12.032+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 111 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:12.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:12.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:12.520+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:12.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:12.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:12.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:12.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:12.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:12.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:13.765 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:13.499+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:13.765 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:13.499+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:13.765 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:13.499+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:13.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:13.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:13.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:14.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:13.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:13.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:13.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:14.775 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:14.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:14.775 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:14.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:14.776 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:14.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:14.776 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:14.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:14.776 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:14.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:14.776 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:14.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:15.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:14.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:14.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:14.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:15.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:15.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:15.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:15.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:15.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:15.531+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:15.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:15.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:15.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:16.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:16.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:16.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:16.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:16.574+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:16.574+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:16.574+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:16.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:16.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:16.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:17.032+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 143 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:17.594+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:17.594+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:17.594+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:17.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:17.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:17.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:17.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:17.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:17.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:18.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:18.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:18.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:18.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:18.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:18.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:18.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:18.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:18.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:19.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:19.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:19.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:19.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:19.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:19.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:19.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:19.593+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:19.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:19.593+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:19.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:19.593+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:19.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:19.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:19.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:19.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:20.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:20.576+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:20.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:20.576+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:20.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:20.576+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:20.750 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:20.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:20.750 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:20.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:20.750 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:20.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:21.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:20.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:21.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:20.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:21.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:20.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:21.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:21.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:21.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:21.552+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:21.553+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:21.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:21.553+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:22.034 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:21.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:22.034 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:21.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:22.034 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:21.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:22.396 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:22.033+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 169 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:22.674 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:22.553+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:22.553+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:22.553+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:22.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:22.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:22.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:23.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:22.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:23.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:22.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:23.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:22.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:23.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:23.530+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:23.530+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:23.530+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:23.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:23.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:23.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:23.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:23.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:23.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:24.551+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:24.551+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:24.551+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:24.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:24.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:24.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:24.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:25.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:24.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:24.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:24.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:25.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:25.599+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:25.599+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:25.599+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:25.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:25.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:25.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:26.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:26.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:26.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:26.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:26.611+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:26.611+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:26.611+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:26.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:26.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:26.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:27.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:27.033+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 196 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:27.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:27.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:27.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:27.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:27.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:27.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:27.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:27.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:28.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:28.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:28.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:28.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:28.683+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:28.683+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:28.683+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:28.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:28.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:28.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:29.825 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:29.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:29.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:29.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:29.639+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:29.639+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:29.639+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:29.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:29.826 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:29.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:29.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:29.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:30.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:30.614+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:30.614+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:30.614+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:30.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:30.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:30.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:30.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:31.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:31.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:31.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:31.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:31.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:31.661+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:31.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:31.661+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:31.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:31.661+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:31.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:31.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:31.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:32.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:32.034+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 223 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:32.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:32.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:32.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:32.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:32.662+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:32.662+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:32.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:32.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:32.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:32.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:33.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:33.695+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:33.695+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:33.695+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:33.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:33.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:33.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:33.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:33.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:33.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:34.722+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:34.722+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:34.722+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:34.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:34.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:34.749 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:34.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:35.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:34.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:34.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:34.748+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:35.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:35.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:35.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:35.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:35.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:35.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:35.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:35.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:35.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:35.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:36.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:36.718+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:36.718+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:36.718+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:36.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:36.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:36.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:36.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:36.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:36.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:37.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:37.034+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 252 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:37.765+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:37.765+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:37.765+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:37.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:37.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:37.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:37.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:37.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:37.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:37.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:38.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:38.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:38.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:38.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:38.731+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:38.731+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:38.731+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:38.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:38.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:38.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:39.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:39.704+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:39.704+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:39.704+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:39.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:39.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:39.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:39.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:39.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:39.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:40.711+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:40.711+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:40.711+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:40.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:40.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:40.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:40.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:41.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:40.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:40.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:40.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:41.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:41.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:41.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:41.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:41.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:41.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:41.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:41.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:42.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:41.761+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:42.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:41.762+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:42.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:41.762+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:42.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:41.763+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:42.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:41.783+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:42.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:41.783+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:42.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:41.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:42.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:41.785+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:42.407 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:42.035+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 279 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:42.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:42.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:42.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:42.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:42.409+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:43.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:42.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:42.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:42.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:42.751+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:42.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:42.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:42.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:43.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:42.730+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:43.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:43.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:43.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:43.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:43.425+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:44.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:43.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:43.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:43.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:43.729+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:43.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:43.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:43.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:43.721+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:44.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:44.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:44.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:44.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:44.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:44.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:44.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:44.701+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:44.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:44.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:44.696+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:44.697+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:45.729 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:45.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:45.729 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:45.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:45.729 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:45.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:45.729 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:45.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:45.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:45.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:45.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:45.741+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:45.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:45.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:45.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:45.729+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:46.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:46.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:46.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:46.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:46.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:46.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:46.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:46.428+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:47.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:46.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:47.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:46.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:47.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:46.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:47.036 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:46.742+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:47.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:46.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:47.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:46.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:47.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:46.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:47.037 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:46.706+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:47.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:47.035+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 308 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:47.699 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:47.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:47.699 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:47.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:47.700 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:47.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:47.700 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:47.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:48.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:47.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:47.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:47.700+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:47.700+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:47.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:47.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:47.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:47.700+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:48.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:48.654+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:48.654+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:48.654+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:48.654+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:48.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:48.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:48.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:48.686+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:48.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:48.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:48.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:48.486+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:49.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:49.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:49.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:49.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:49.696+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:49.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:49.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:49.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:49.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:49.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:49.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:49.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:49.648+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:50.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:50.660+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:50.660+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:50.660+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:50.661+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:50.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:50.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:50.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:50.500+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:50.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:50.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:50.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:50.605+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:51.662+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:51.662+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:51.662+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:51.662+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:51.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:51.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:51.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:51.467+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:51.593+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:51.593+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:51.593+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:51.594+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:52.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:52.035+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 335 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:52.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:52.673+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:52.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:52.673+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:52.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:52.673+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:52.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:52.673+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:52.488+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:52.488+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:52.488+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:52.489+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:52.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:52.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:52.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:52.611+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:53.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:53.659+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:53.659+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:53.659+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:53.659+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:53.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:53.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:53.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:53.569+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:53.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:53.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:53.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:53.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:54.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:54.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:54.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:54.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:54.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:54.641+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:54.641+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:54.641+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:54.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:54.642+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:54.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:55.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:55.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:55.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:55.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:55.501+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:55.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:55.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:55.610+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:55.611+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:55.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:55.627+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:55.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:55.627+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:55.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:55.627+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:55.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:55.628+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:56.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:56.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:56.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:56.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:56.632+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:56.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:56.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:56.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:56.498+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:56.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:56.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:56.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:56.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:56.572+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:57.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:29:57.036+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 365 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:29:57.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:57.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:57.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:57.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:57.512+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:57.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:57.592+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:57.592+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:57.592+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:57.593+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:57.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:57.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:57.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:57.601+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:58.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:58.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:58.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:58.629+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:58.630+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:58.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:58.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:58.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:58.554+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:58.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:58.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:58.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:58.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:59.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:59.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:59.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:29:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:29:59.507+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:59.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:29:59.666+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:59.678+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:59.678+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:59.678+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:29:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:29:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:29:59.679+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:00.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:00.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:00.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:00.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:00.729+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:00.729+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:00.729+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:00.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:00.729+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:00.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:00.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:00.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:00.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:01.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:01.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:01.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:01.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:01.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:01.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:01.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:01.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:01.749+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:01.682+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:01.682+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:01.682+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:01.683+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:02.333 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:02.036+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 394 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:02.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:02.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:02.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:02.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:02.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:02.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:02.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:02.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:02.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:02.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:02.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:02.636+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:02.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:02.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:02.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:02.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:02.738+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:03.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:03.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:03.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:03.588+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:03.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:03.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:03.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:03.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:03.738+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:03.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:03.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:03.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:03.499+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:04.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:04.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:04.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:04.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:04.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:04.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:04.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:04.765 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:04.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:05.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:04.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:04.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:04.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:04.764+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:05.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:05.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:05.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:05.491+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:05.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:05.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:05.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:05.532+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:05.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:05.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:05.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:05.795+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:06.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:06.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:06.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:06.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:06.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:06.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:06.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:06.469+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:06.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:06.495+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:06.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:06.495+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:06.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:06.495+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:06.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:06.496+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:07.159 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:07.037+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 421 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:06.809+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:06.809+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:06.809+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:06.810+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:07.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:07.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:07.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:07.514+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:07.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:07.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:07.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:07.851 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:07.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:07.851 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:07.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:08.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:07.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:07.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:07.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:07.850+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:08.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:08.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:08.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:08.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:08.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:08.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:08.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:08.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:08.491+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:09.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:08.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:08.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:08.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:08.875+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:09.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:09.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:09.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:09.506+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:09.489+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:09.489+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:09.489+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:09.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:09.490+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:09.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:09.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:09.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:09.865+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:10.909 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:10.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:10.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:10.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:10.477+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:10.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:10.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:10.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:10.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:10.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:11.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:10.909+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:10.909+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:10.909+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:10.909+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:11.907 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:11.431+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:11.907 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:11.431+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:11.907 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:11.431+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:11.907 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:11.431+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:11.908 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:11.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:11.908 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:11.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:11.908 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:11.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:11.908 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:11.499+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:12.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:11.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:12.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:11.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:12.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:11.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:12.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:11.907+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:12.161 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:12.038+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 449 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:12.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:12.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:12.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:12.463+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:12.456+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:12.456+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:12.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:12.456+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:12.894 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:12.457+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:13.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:12.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:12.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:12.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:12.893+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:13.886 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:13.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:13.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:13.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:13.487+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:13.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:13.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:13.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:13.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:13.507+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:14.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:13.885+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:13.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:13.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:13.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:14.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:14.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:14.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:14.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:14.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:14.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:14.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:14.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:14.922 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:14.463+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:15.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:14.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:14.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:14.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:14.921+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:15.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:15.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:15.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:15.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:15.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:15.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:15.466+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:15.467+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:16.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:15.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:15.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:15.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:15.965+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:16.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:16.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:16.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:16.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:16.465+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:16.465+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:16.465+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:16.465+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:17.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:17.038+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 479 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:17.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:16.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:17.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:16.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:17.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:16.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:17.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:16.957+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:17.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:17.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:17.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:17.458+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:17.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:17.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:17.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:17.457+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:17.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:17.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:17.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:17.970+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:18.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:18.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:18.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:18.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:18.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:18.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:18.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:18.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:18.453+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:19.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:19.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:19.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:19.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:18.978+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:19.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:19.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:19.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:19.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:19.413+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:19.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:19.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:19.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:19.437+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:20.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:20.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:20.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:20.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:20.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:20.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:20.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:20.014+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:20.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:20.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:20.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:20.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:20.427+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:20.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:20.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:20.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:20.422+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:21.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.012+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:21.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:21.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:21.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:21.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:21.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:21.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:21.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:21.376+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:21.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:21.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:21.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:21.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:21.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:21.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:21.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:21.417+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:22.386 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:22.039+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 506 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:22.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:22.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:22.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:22.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:21.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:22.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:22.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:22.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:22.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:22.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:22.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:22.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:22.404+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:23.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:23.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:23.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:23.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:23.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:23.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:23.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:23.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:23.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:23.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:23.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:23.434+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:23.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:23.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:23.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:23.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:23.396+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:24.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:24.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:24.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:24.056+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:24.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:24.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:24.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:24.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:24.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:24.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:24.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:24.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:24.372+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:25.383 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:25.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:25.383 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:25.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:25.383 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:25.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:25.383 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:25.079+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:25.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:25.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:25.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:25.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:25.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:25.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:25.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:25.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:25.383+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:26.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:26.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:26.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:26.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:26.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:26.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:26.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:26.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:26.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:26.344+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:26.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:26.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:26.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:26.520+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:27.320 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:27.040+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 533 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:27.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:27.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:27.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:27.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:27.321 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:27.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:27.321 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:27.062+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:27.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:27.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:27.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:27.497+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:27.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:27.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:27.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:27.321+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:28.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:28.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:28.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:28.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:28.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:28.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:28.349 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:28.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:28.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:28.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:28.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:28.513+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:28.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:28.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:28.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:28.348+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:29.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:29.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:29.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:29.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:29.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:29.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:29.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:29.040+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:29.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:29.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:29.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:29.322+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:29.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:29.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:29.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:29.467+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:30.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:30.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:30.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:30.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:30.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:30.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:30.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:30.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:30.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:30.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:30.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:30.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:30.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:30.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:30.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:30.294+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:31.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:31.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:31.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:31.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:31.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:31.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:31.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:31.026+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:31.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:31.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:31.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:31.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:31.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:31.447+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:31.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:31.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:31.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:31.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:31.324+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:32.326 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:32.040+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 560 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:32.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:32.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:32.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:32.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:32.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:32.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:32.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:32.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:32.445+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:32.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:32.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:32.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:32.326+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:33.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:33.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:33.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:33.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:32.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:33.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:33.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:33.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:33.372+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:33.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:33.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:33.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:33.424+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:34.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:34.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:34.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:34.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:34.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:34.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:34.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:34.472+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:34.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:34.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:34.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:34.412+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:35.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:35.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:35.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:35.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:34.993+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:35.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:35.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:35.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:35.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:35.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:35.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:35.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:35.368+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:36.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:36.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:36.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:36.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.003+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:36.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:36.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:36.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:36.484+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:36.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:36.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:36.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:36.398+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:37.404 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:37.040+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 589 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:37.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:37.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:37.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:37.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:36.958+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:37.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:37.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:37.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:37.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:37.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:37.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:37.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:37.404+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:38.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:37.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:37.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:37.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:37.992+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:38.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:38.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:38.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:38.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:38.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:38.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:38.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:38.428+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:38.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:38.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:38.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:38.456+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:39.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:39.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:39.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:39.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:39.042+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:39.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:39.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:39.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:39.478+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:39.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:39.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:39.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:39.426+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:40.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:40.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:40.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:40.522+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:40.455+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:40.455+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:40.455+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:40.456+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:41.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.968+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.968+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.968+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:40.968+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:41.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:41.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:41.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:41.531+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:41.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:41.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:41.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:41.448+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:42.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:42.041+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 616 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:42.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:41.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:42.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:41.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:42.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:41.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:42.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:41.926+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:42.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:42.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:42.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:42.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:42.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:42.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:42.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:42.468+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:42.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:42.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:43.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:42.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:42.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:42.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:42.959+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:43.441+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:43.441+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:43.441+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:43.442+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:43.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:43.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:43.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:43.609+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:44.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:43.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:43.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:43.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:43.950+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:44.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:44.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:44.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:44.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:44.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:44.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:44.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:44.634+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:44.906 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:44.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:44.906 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:44.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:44.906 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:44.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:44.906 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:44.454+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:45.174 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:44.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:45.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:44.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:45.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:44.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:45.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:44.905+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:45.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:45.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:45.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:45.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:45.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:45.481+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:45.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:45.482+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:45.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:45.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:45.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:45.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:45.595+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:46.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:45.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:45.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:45.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:45.946+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:46.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:46.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:46.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:46.435+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:46.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:46.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:46.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:46.610+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:47.041+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 643 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:46.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:46.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:46.969+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:46.970+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:47.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:47.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:47.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:47.428+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:47.429+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:47.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:47.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:47.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:47.571+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:48.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:48.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:48.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:48.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:48.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:48.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:48.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:48.382+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:48.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:48.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:48.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:48.567+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:49.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:49.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:49.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:49.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:48.964+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:49.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:49.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:49.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:49.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:49.603+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:49.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:49.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:49.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:49.356+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:50.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:49.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:50.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:49.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:50.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:49.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:50.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:49.963+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:50.633 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:50.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:50.633 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:50.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:50.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:50.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:50.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:50.309+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:50.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:50.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:50.632+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:50.633+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:51.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:50.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:51.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:50.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:51.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:50.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:51.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:50.955+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:51.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:51.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:51.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:51.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:51.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:51.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:51.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:51.330+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:51.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:51.614+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:52.355 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:52.042+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 670 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:52.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:51.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:52.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:51.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:52.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:51.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:52.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:51.974+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:52.637 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:52.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:52.637 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:52.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:52.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:52.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:52.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:52.355+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:52.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:52.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:52.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:52.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:52.637+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:53.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:52.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:53.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:52.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:53.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:52.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:53.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:52.997+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:53.632 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:53.632 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:53.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:53.382+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:53.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:53.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:53.631+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:53.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:53.631+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:54.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:53.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:54.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:53.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:54.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:53.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:54.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:53.991+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:54.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:54.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:54.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:54.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:54.359+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:54.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:54.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:54.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:54.585+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:55.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:55.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:55.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:55.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:55.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:55.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:55.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:55.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:55.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:55.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:55.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:55.553+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:55.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:55.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:55.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:55.313+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:56.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:56.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:56.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:56.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:56.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:56.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:56.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:56.056+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:56.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:56.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:56.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:56.345+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:56.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:56.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:56.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:56.529+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:57.332 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:30:57.042+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 699 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:30:57.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:57.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:57.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:57.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:57.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:57.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:57.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:57.008+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:57.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:57.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:57.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:57.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:57.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:57.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:57.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:57.631 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:57.332+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:58.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:58.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:58.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:58.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:58.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:58.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:58.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:58.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:58.301+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:58.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:58.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:58.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:58.472+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:59.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:59.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:59.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:59.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:59.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:59.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:30:59.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:30:59.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:59.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:59.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:59.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:59.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:30:59.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:59.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:59.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:59.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:30:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:30:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:30:59.323+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:00.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:00.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:00.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:00.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:00.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:00.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:00.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:00.042+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:00.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:00.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:00.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:00.480+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:00.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:00.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:00.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:00.373+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:01.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:01.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:01.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:01.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:01.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:01.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:01.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:01.048+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:01.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:01.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:01.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:01.468+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:01.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:01.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:01.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:01.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:01.388+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:02.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:02.043+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 728 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:02.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:02.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:02.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:02.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:02.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:02.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:02.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:02.098+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:02.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:02.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:02.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:02.442+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:02.401+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:02.401+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:02.401+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:02.402+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:03.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:03.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:03.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:03.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:03.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:03.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:03.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:03.095+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:03.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:03.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:03.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:03.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:03.417+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:03.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:03.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:03.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:03.379+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:04.375 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:04.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:04.375 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:04.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:04.375 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:04.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:04.375 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:04.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:04.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:04.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:04.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:04.458+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:04.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:04.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:04.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:04.375+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:05.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:05.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:05.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:05.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:05.122+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:05.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:05.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:05.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:05.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:05.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:05.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:05.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:05.379+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:06.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:06.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:06.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:06.383+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:06.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:06.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:06.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:06.154+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:06.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:06.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:06.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:06.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:06.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:06.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:06.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:06.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:07.378 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:07.044+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 767 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:07.378 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:07.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:07.378 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:07.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:07.378 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:07.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:07.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:07.128+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:07.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:07.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:07.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:07.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:07.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:07.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:07.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:07.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:07.378+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:08.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:08.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:08.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:08.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:08.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:08.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:08.379+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:08.373+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:08.373+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:08.373+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:08.373+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:08.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:08.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:08.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:08.153+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:09.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:09.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:09.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:09.405+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:09.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:09.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:09.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:09.396+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:09.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:09.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:09.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:09.176+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:10.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:10.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:10.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:10.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:10.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:10.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:10.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:10.143+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:10.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:10.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:10.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:10.437+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:10.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:10.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:10.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:10.396+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:11.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:11.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:11.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:11.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:11.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:11.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:11.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:11.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:11.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:11.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:11.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:11.474+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:11.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:11.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:11.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:11.413+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:12.424 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:12.044+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 794 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:12.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:12.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:12.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:12.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:12.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:12.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:12.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:12.448+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:12.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:13.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:13.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:13.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:13.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:13.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:13.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:13.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:13.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:13.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:13.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:13.491+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:13.491+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:13.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:13.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:13.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:13.487+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:14.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:14.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:14.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:14.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:14.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:14.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:14.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:14.024+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:14.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:14.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:14.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:14.505+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:14.500+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:14.500+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:14.500+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:14.501+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:15.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:15.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:15.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:15.060+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:15.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:15.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:15.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:15.522+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:15.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:15.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:15.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:15.454+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:16.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:16.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:16.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:16.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:16.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:16.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:16.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:16.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:16.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:16.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:16.434+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:16.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:16.435+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:17.394 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:17.044+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 823 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:17.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:17.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:17.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:17.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:17.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:17.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:17.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:17.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:17.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:17.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:17.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:17.489+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:17.393+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:17.393+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:17.393+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:17.394+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:18.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:18.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:18.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:18.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:18.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:18.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:18.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:18.030+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:18.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:18.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:18.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:18.445+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:18.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:18.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:18.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:18.413+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:19.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:19.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:19.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:19.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:19.008+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:19.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:19.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:19.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:19.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:19.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:19.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:19.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:19.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:19.460+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:20.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:20.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:20.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:20.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:20.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:20.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:20.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:20.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:20.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:20.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:20.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:20.402+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:20.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:20.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:20.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:20.470+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:21.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:21.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:21.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:21.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:21.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:21.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:21.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:21.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:21.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:21.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:21.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:21.417+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:21.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:21.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:21.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:21.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:21.505+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:22.045+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 850 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:22.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:22.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:22.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:22.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:22.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:22.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:22.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:22.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:22.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:22.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:22.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:22.514+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:22.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:22.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:22.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:22.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:22.426+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:23.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:23.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:23.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:23.078+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:23.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:23.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:23.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:23.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:23.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:23.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:23.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:23.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:24.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:24.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:24.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:24.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:24.488+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:24.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:24.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:24.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:24.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:24.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:25.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:25.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:25.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:25.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:25.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:25.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:25.087+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:25.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:25.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:25.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:25.442+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:25.602+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:26.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:26.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:26.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:26.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:26.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:26.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:26.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:26.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:26.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:26.614+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:27.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:27.045+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 877 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:27.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:27.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:27.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:27.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:27.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:27.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:27.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:27.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:27.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:27.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:27.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:27.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:27.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:27.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:27.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:27.574+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:27.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:27.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:27.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:27.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:27.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:27.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:27.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:27.469+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:28.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:28.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:28.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:28.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:28.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:28.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:28.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:28.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:28.612+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:28.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:28.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:28.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:28.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:29.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:29.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:29.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:29.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:29.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:29.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:29.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:29.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:29.579+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:29.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:29.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:29.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:29.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:29.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:29.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:29.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:29.504+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:30.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:30.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:30.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:30.020+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:30.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:30.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:30.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:30.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:30.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:30.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:30.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:30.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:30.551+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:31.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:31.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:31.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:31.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:31.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:31.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:31.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:31.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:31.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:31.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:31.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:31.536+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:32.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:32.046+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 904 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:32.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:32.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:32.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:32.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:32.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:32.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:32.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:32.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:32.541+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:33.391 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:33.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:33.391 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:33.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:33.391 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:33.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:33.391 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:33.020+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:33.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:33.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:33.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:33.391+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:33.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:33.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:33.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:33.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:33.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:33.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:33.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:33.582+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:34.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:34.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:34.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:34.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:34.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:34.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:34.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:34.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:34.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:34.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:34.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:34.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:34.550+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:35.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:35.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:35.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:35.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:35.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:35.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:35.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:35.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:35.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:35.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:35.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:35.420+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:35.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:35.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:35.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:35.511+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:36.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:36.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:36.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:36.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:36.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:36.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:36.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:36.116+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:36.512+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:36.512+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:36.512+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:36.513+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:36.387+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:36.387+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:36.387+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:36.388+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:37.341 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:37.046+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 933 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:37.341 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:37.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:37.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:37.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:37.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:37.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:37.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:37.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:37.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:37.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:37.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:37.536+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:37.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:37.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:37.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:37.341+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:38.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:38.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:38.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:38.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:38.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:38.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:38.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:38.294+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:38.295+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:38.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:38.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:38.560+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:38.561+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:39.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:39.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:39.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:39.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:39.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:39.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:39.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:39.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:39.327+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:39.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:39.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:39.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:39.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:40.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:40.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:40.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:40.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:40.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:40.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:40.309+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:40.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:40.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:40.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:40.637+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:41.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:41.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:41.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:41.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:41.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:41.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:41.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:41.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:41.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:41.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:41.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:41.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:41.678+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:42.047+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 962 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:42.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:42.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:42.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:42.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:42.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:42.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:42.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:42.086+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:42.632+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:42.632+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:42.632+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:42.633+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:43.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:43.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:43.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:43.229+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:43.047+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:43.654+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:44.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:44.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:44.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:44.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:44.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:44.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:44.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:44.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:44.272+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:44.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:44.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:44.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:44.635+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:45.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:45.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:45.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:45.233+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:45.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:45.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:45.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:45.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:45.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:45.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:45.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:45.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:46.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:46.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:46.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:46.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:46.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:46.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:46.241+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:46.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:46.241+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:46.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:46.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:46.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:46.625+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:47.047+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1005 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:47.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:47.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:47.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:47.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:47.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:47.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:47.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:47.264+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:47.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:47.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:47.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:47.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:48.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:48.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:48.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:48.087+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:48.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:48.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:48.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:48.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:48.224+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:48.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:48.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:48.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:48.624+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:49.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:49.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:49.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:49.134+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:49.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:49.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:49.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:49.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:49.182+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:49.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:49.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:49.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:49.576+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:50.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:50.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:50.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:50.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:50.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:50.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:50.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:50.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:50.205+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:50.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:50.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:50.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:50.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:50.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:50.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:50.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:50.611+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:51.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:51.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:51.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:51.253+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:51.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:51.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:51.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:51.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:51.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:51.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:51.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:51.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:52.048+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1048 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:52.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:52.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:52.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:52.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:52.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:52.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:52.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:52.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:52.289+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:52.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:52.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:52.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:52.546+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:53.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:53.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:53.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:53.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:53.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:53.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:53.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:53.160+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:53.588 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:53.588 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:53.588 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:53.588 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:53.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:53.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:53.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:53.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:53.588+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:54.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:54.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:54.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:54.128+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:54.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:54.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:54.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:54.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:54.344+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:54.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:54.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:54.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:54.576+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:55.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:55.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:55.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:55.148+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:55.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:55.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:55.357+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:55.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:55.358+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:55.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:55.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:55.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:55.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:55.600+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:56.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:56.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:56.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:56.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:56.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:56.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:56.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:56.161+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:56.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:56.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:56.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:56.331+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:56.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:56.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:56.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:56.571+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:57.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:31:57.048+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1094 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:31:57.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:57.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:57.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:57.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:57.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:57.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:57.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:57.145+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:57.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:57.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:57.312+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:57.313+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:57.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:57.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:57.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:57.555+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:58.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:58.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:58.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:58.347+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:58.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:58.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:58.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:58.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:58.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:58.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:58.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:58.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:58.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:58.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:58.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:58.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:58.519+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:59.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:59.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:59.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:59.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:59.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:59.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:31:59.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:31:59.131+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:59.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:59.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:59.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:31:59.335+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:59.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:59.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:59.513+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:31:59.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:31:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:31:59.514+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:00.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:00.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:00.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:00.134+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:00.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:00.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:00.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:00.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:00.342+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:00.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:00.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:00.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:00.518+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:01.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:01.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:01.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:01.158+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:01.349+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:01.349+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:01.349+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:01.350+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:01.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:01.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:01.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:01.552+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:02.317 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:02.049+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1137 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:02.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:02.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:02.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:02.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:02.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:02.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:02.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:02.153+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:02.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:02.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:02.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:02.317+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:02.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:02.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:02.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:02.505+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:03.317+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:03.317+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:03.317+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:03.318+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:03.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:03.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:03.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:03.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:03.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:03.495 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:03.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:03.495 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:03.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:03.924 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:03.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:03.493+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:04.143+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:04.143+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:04.143+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:04.143+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:04.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:04.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:04.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:04.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:04.336+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:04.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:04.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:04.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:04.448+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:05.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:05.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:05.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:05.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:05.398+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:05.398+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:05.398+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:05.399+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:05.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:05.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:05.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:05.330+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:06.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:06.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:06.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:06.376+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:06.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:06.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:06.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:06.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:06.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:06.361+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:07.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:07.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:07.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:07.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:07.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:07.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:07.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:07.197+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:07.336 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:07.050+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1181 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:07.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:07.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:07.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:07.335+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:07.421+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:07.421+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:07.421+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:07.421+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:08.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:08.381+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:08.382+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:08.382+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:08.383+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:08.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:08.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:08.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:08.423+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:09.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:09.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:09.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:09.245+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:09.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:09.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:09.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:09.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:09.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:09.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:09.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:09.455+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:10.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:10.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:10.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:10.278+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:10.410+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:10.410+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:10.410+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:10.410+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:10.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:10.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:10.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:10.403+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:11.640 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:11.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:11.640 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:11.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:11.640 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:11.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:11.640 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:11.268+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:11.640 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:11.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:11.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:11.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:11.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:11.641 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:11.371+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:12.390 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:12.050+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1225 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:12.390 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:12.226+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:12.390 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:12.226+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:12.390 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:12.226+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:12.390 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:12.227+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:12.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:12.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:12.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:12.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:12.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:12.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:12.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:12.389+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:13.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:13.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:13.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:13.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:13.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:13.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:13.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:13.225+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:13.519 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:13.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:13.519 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:13.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:13.519 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:13.435+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:13.519 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:13.436+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:13.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:13.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:13.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:13.518+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:14.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:14.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:14.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:14.179+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:14.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:14.430 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:14.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:14.431 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:14.416+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:14.431 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:14.417+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:14.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:14.977 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:14.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:14.977 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:14.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:14.977 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:14.513+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:15.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:15.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:15.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:15.152+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:15.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:15.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:15.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:15.487+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:15.439+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:15.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:15.439+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:15.439+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:15.439+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:16.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:16.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:16.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:16.391+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:16.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:16.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:16.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:16.157+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:16.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:16.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:16.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:16.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:17.423 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:17.051+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1270 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:17.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:17.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:17.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:17.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:17.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:17.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:17.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:17.131+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:17.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:17.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:17.422+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:17.423+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:17.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:17.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:17.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:17.465+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:18.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:32:18.299+0000 7fe63398c700 0 monclient(hunting): authenticate timed out after 300 2024-01-19T06:32:18.301 INFO:teuthology.orchestra.run.smithi134.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-19T06:32:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:18.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:18.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:18.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:18.117+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:18.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:18.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:18.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:18.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:18.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:18.696 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:18.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:18.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:18.461+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:18.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:18.461+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:19.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-01-19T06:32:19.407 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:32:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:19.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:19.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:19.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:19.119+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:19.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:19.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:19.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:19.791 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:19.467+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:19.792 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:19.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:19.792 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:19.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:19.792 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:19.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:19.792 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:19.441+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:20.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:20.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:20.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:20.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:20.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:20.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:20.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:20.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:20.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:20.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:20.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:20.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:20.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:20.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:20.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:20.400+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:21.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:21.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:21.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:21.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:21.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:21.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:21.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:21.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:21.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:21.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:21.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:21.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:21.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:21.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:21.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:21.353+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:22.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:22.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:22.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:22.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:22.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:22.049+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:22.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:22.050+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:22.370 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:22.051+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1312 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:22.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:22.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:22.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:22.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:22.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:22.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:22.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:22.369+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:23.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:23.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:23.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:23.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:23.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:23.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:23.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:23.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:23.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:23.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:23.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:23.444+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:23.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:23.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:23.645 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:23.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:23.646 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:23.348+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:24.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:24.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:24.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:24.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:24.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:24.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:24.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:24.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:24.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:24.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:24.328+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:24.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:24.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:24.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:24.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:25.326 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:25.326 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:25.326 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:25.326 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:24.993+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:25.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:25.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:25.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:25.326+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:25.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:25.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:25.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:25.479+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:25.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:25.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:26.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:25.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:26.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:25.949+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:26.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:26.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:26.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:26.509+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:26.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:26.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:26.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:26.366+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:27.340 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:27.052+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1356 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:27.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:26.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:27.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:26.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:27.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:26.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:27.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:26.994+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:27.636 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:27.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:27.636 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:27.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:27.636 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:27.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:27.636 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:27.339+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:27.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:27.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:27.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:27.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:27.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:27.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:27.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:27.552+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:28.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:28.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:28.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:28.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:28.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:28.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:28.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:28.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:28.552+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:28.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:28.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:28.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:28.355+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:29.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:29.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:29.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:29.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:29.048+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:29.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:29.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:29.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:29.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:29.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:29.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:29.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:29.355+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:30.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:30.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:30.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:30.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:30.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:30.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:30.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:30.060+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:30.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:30.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:30.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:30.522+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:30.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:30.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:30.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:30.384+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:31.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:31.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:31.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:31.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:31.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:31.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:31.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:31.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:31.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:31.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:31.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:31.480+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:31.369+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:31.369+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:31.369+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:31.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:31.370+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:32.356 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:32.052+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1399 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:32.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:32.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:32.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:32.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:32.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:32.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:32.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:32.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:32.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:32.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:32.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:32.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:32.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:32.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:32.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:32.356+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:33.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:33.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:33.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:33.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:33.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:33.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:33.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:33.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:33.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:33.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:33.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:33.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:33.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:33.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:33.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:33.376+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:34.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:34.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:34.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:34.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:34.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:34.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:34.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:34.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:34.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:34.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:34.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:34.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:34.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:34.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:34.379+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:34.380+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:35.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:35.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:35.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:35.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:35.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:35.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:35.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:35.068+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:35.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:35.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:35.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:35.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:35.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:35.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:35.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:35.334+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:35.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:35.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:35.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:35.594+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:36.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:36.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:36.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:36.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:36.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:36.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:36.325 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:36.106+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:36.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:36.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:36.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:36.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:36.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:36.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:36.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:36.324+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:37.344 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:37.054+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1504 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:37.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:37.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:37.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:37.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:37.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:37.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:37.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:37.108+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:37.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:37.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:37.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:37.345+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:37.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:37.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:37.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:37.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:37.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:38.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:38.309+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:38.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:38.309+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:38.309+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:38.309+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:38.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:38.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:38.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:38.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:38.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:38.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:38.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:38.574+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:39.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:39.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:39.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:39.347+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:39.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:39.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:39.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:39.165+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:39.623+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:39.623+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:39.623+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:39.624+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:40.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:40.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:40.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:40.315+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:40.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:40.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:40.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:40.117+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:40.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:40.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:40.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:40.620+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:41.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:41.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:41.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:41.081+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:41.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:41.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:41.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:41.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:41.287+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:41.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:41.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:41.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:41.577+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:42.054+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1596 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:42.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:42.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:42.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:42.283+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:42.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:42.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:42.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:42.119+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:42.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:42.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:42.603+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:42.604+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:43.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:43.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:43.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:43.132+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:43.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:43.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:43.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:43.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:43.269+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:43.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:43.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:43.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:43.593+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:44.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:44.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:44.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:44.084+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:44.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:44.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:44.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:44.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:44.227+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:44.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:44.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:44.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:44.638+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:45.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:45.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:45.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:45.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:45.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:45.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:45.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:45.202+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:45.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:45.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:45.616+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:45.617+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:46.113+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:46.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:46.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:46.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:46.209+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:46.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:46.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:46.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:46.640+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:47.054+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1690 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:47.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:47.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:47.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:47.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:47.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:47.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:47.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:47.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:47.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:47.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:47.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:47.659+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:48.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:48.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:48.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:48.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:48.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:48.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:48.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:48.136+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:49.081 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:48.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.082 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:48.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.082 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:48.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.082 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:48.676+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:49.136+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:49.136+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:49.136+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:49.137+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:49.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:49.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:49.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:49.081+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:49.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:49.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:49.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:49.654+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:50.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:50.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:50.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:50.103+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:50.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:50.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:50.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:50.095+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:50.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:50.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:50.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:50.664+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:51.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:51.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:51.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:51.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:51.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:51.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:51.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:51.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:51.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:51.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:51.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:51.675+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:52.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:52.055+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1781 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:52.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:52.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:52.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:52.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:52.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:52.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:52.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:52.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:52.156+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:53.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:52.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:52.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:52.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.130 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:52.676+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:53.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:53.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:53.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:53.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:53.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:53.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:53.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:53.129+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:53.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:53.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:53.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:53.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:53.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:53.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:53.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:53.683+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:54.365 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:54.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:54.365 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:54.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:54.365 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:54.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:54.365 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:54.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:54.366 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:54.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:54.366 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:54.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:54.366 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:54.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:54.366 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:54.093+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:55.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:54.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:54.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:54.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.098 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:54.679+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:55.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:55.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:55.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:55.097+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:55.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:55.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:55.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:55.221+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.932 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.932 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:55.932 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:55.643+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:56.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:56.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:56.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:56.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:56.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:56.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:56.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:56.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:56.132+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:56.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:56.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:56.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:56.634+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:57.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:32:57.055+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1877 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:32:57.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:57.202+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:57.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:57.202+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:57.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:57.202+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:57.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:57.203+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:57.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:57.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:57.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:57.154+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:57.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:57.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:57.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:57.587+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:58.211 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:58.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:58.211 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:58.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:58.211 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:58.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:58.211 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:58.167+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:58.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:58.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:58.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:58.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:58.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:58.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:58.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:58.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:58.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:58.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:58.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:58.637+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:59.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:59.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:59.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:32:59.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:59.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:59.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:59.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:59.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:32:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:32:59.135+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:32:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:59.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:59.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:59.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:32:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:32:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:32:59.642+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:00.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:00.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:00.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:00.166+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:00.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:00.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:00.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:00.125+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:00.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:00.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:00.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:00.606+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:01.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:01.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:01.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:01.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:01.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:01.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:01.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:01.159+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:01.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:01.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:01.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:01.601+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:02.056+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 1969 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:02.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:02.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:02.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:02.228+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:02.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:02.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:02.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:02.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:02.204+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:02.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:02.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:02.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:02.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:02.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:02.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:02.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:02.625+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:03.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:03.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:03.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:03.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:03.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:03.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:03.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:03.235+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:03.534 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:03.228+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:03.534 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:03.228+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:03.534 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:03.228+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:03.534 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:03.229+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:03.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:03.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:03.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:03.601+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:04.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:04.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:04.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:04.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:04.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:04.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:04.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:04.195+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:04.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:04.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:04.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:04.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:04.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:04.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:04.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:04.184+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:04.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:04.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:04.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:04.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:05.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:05.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:05.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:05.191+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:05.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:05.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:05.189+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:05.596 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:05.189+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:05.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:05.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:05.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:05.596+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:06.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:06.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:06.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:06.202+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:06.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:06.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:06.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:06.163+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:06.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:06.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:06.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:06.563+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:07.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:07.057+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2064 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:07.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:07.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:07.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:07.191+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:07.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:07.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:07.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:07.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:07.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:07.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:07.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:07.607+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:08.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:08.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:08.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:08.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:08.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:08.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:08.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:08.206+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:08.626 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:08.204+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:08.626 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:08.204+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:08.626 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:08.204+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:08.626 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:08.205+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:08.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:08.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:08.624+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:08.625+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:09.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:09.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:09.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:09.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:09.227+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:09.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:09.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:09.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:09.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:09.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:09.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:09.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:09.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:09.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:09.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:09.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:09.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:09.578+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:10.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:10.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:10.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:10.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:10.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:10.231+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:10.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:10.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:10.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:10.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:10.211+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:10.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:10.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:10.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:10.557+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:11.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:11.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:11.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:11.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:11.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:11.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:11.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:11.261+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:11.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:11.191+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:11.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:11.191+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:11.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:11.191+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:11.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:11.192+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:11.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:11.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:11.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:11.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:12.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:12.057+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2157 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:12.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:12.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:12.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:12.165+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:12.265+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:12.265+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:12.265+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:12.265+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:12.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:12.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:12.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:12.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:12.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:12.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:12.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:12.505+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:13.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:13.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:13.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:13.211+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:13.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:13.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:13.516 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:13.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:13.517 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:13.289+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:13.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:13.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:13.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:13.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:14.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:14.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:14.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:14.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:14.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:14.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:14.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:14.248+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:14.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:14.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:14.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:14.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:14.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:14.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:14.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:14.270+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:14.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:14.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:14.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:14.558+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:15.559 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:15.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:15.559 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:15.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:15.559 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:15.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:15.559 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:15.270+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:15.559 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:15.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:15.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:15.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:15.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:15.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:15.560 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:15.255+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:15.869 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:15.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:15.869 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:15.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:15.869 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:15.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:15.869 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:15.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:16.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:16.258+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:16.258+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:16.258+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:16.259+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:16.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:16.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:16.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:16.565 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:16.305+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:16.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:16.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:16.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:16.564+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:17.348 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:17.058+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2255 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:17.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:17.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:17.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:17.349 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:17.284+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:17.601 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:17.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:17.601 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:17.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:17.601 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:17.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:17.601 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:17.348+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:17.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:17.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:17.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:17.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:17.600+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:18.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:18.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:18.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:18.295+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:18.373+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:18.373+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:18.623 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:18.373+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:18.624 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:18.374+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:18.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:18.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:18.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:18.622+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:19.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:19.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:19.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:19.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:19.614 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:19.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:19.615 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:19.336+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:19.615 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:19.615 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:19.615 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:19.615 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:19.307+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:19.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:19.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:19.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:19.614+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:20.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:20.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:20.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:20.288+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:20.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:20.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:20.571 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:20.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:20.572 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:20.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:20.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:20.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:20.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:20.571+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:21.646 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:21.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:21.646 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:21.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:21.646 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:21.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:21.646 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:21.319+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:21.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:21.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:21.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:21.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:21.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:21.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:21.647 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:21.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:21.648 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:21.309+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:22.325 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:22.060+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2347 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:22.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:22.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:22.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:22.341+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:22.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:22.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:22.325+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:22.325+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:22.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:22.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:22.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:22.553+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:23.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:23.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:23.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:23.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:23.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:23.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:23.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:23.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:23.350+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:23.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:23.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:23.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:23.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:23.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:24.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:24.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:24.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:24.325+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:24.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:24.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:24.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:24.484+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:24.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:24.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:24.347+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:24.347+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:25.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:25.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:25.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:25.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:25.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:25.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:25.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:25.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:25.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:25.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:25.464+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:25.465+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:25.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:25.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:25.352+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:25.353+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:26.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:26.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:26.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:26.239+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:26.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:26.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:26.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:26.479+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:26.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:26.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:26.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:26.375+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:27.387 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:27.059+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2441 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:27.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:27.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:27.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:27.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:27.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:27.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:27.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:27.199+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:27.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:27.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:27.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:27.386+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:27.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:27.438+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:27.438+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:27.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:28.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:28.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:28.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:28.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:28.207+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:28.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:28.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:28.430+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:28.483 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:28.431+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:28.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:28.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:28.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:28.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:29.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:29.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:29.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:29.211+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:29.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:29.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:29.476 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:29.447+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:29.477 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:29.448+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:29.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:29.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:29.474+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:29.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:30.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:30.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:30.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:30.261+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:30.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:30.462+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:30.462+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:30.462+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:30.463+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:30.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:30.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:30.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:30.484+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:31.647 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:31.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:31.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:31.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:31.476+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:31.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:31.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:31.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:31.649 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:31.286+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:31.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:31.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:31.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:31.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:31.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:31.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:31.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:31.448+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:32.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:32.060+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2534 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:32.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:32.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:32.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:32.274+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:32.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:32.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:32.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:32.442+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:32.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:32.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:32.517+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:32.518+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:33.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:33.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:33.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:33.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:33.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:33.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:33.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:33.276+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:33.536 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:33.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:33.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:33.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:33.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:33.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:33.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:33.401+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:33.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:33.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:33.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:33.541+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:34.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:34.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:34.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:34.260+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:34.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:34.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:34.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:34.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:34.373+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:34.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:34.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:34.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:34.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:34.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:35.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:35.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:35.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:35.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:35.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:35.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:35.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:35.299+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:35.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:35.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:35.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:35.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:35.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:35.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:35.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:35.418+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:35.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:35.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:35.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:35.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:35.629+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:36.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:36.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:36.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:36.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:36.309+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:36.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:36.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:36.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:36.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:36.406+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:36.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:36.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:36.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:36.629+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:37.323 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:37.060+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2632 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:37.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:37.374+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:37.374+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:37.374+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:37.375+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:37.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:37.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:37.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:37.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:37.323+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:37.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:37.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:37.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:37.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:37.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:37.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:37.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:37.620+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:38.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:38.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:38.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:38.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:38.277+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:38.605 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:38.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:38.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:38.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:38.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:38.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:38.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:38.353+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:38.604+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:38.604+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:38.604+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:38.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:38.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:39.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:39.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:39.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:39.248+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:39.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:39.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:39.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:39.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:39.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:39.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:39.310+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:39.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:39.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:39.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:39.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:39.651+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:40.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:40.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:40.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:40.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:40.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:40.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:40.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:40.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:40.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:40.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:40.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:40.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:40.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:40.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:40.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:40.299+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:40.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:40.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:40.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:40.641+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:41.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:41.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:41.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:41.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:41.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:41.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:41.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:41.236+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:41.648 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:41.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:41.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:41.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:41.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:41.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:41.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:41.285+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:41.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:41.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:41.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:41.680+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:42.061+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2726 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:42.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:42.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:42.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:42.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:42.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:42.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:42.259+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:42.260+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:42.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:42.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:42.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:42.650+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:43.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:43.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:43.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:43.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:43.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:43.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:43.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:43.538 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:43.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:43.538 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:43.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:43.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:43.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:43.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:43.693+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:44.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:44.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:44.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:44.323+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:44.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:44.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:44.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:44.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:44.321+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:44.692+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:45.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:45.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:45.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:45.284+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:45.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:45.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:45.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:45.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:45.294+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:45.668+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:45.668+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:45.668+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:45.669+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:46.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:46.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:46.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:46.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:46.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:46.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:46.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:46.266+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:46.589 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:46.276+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:46.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:46.276+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:46.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:46.276+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:46.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:46.278+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:46.694+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:46.694+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:46.694+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:46.695+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:47.062+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2822 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:47.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:47.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:47.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:47.309+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:47.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:47.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:47.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:47.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:47.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:47.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:47.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:47.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:47.662+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:48.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:48.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:48.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:48.278+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:48.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:48.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:48.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:48.638 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:48.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:48.638 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:48.291+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:48.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:48.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:48.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:48.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:48.637+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:49.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:49.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:49.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:49.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:49.266+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:49.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:49.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:49.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:49.630 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:49.278+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:49.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:49.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:49.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:49.629+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:50.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:50.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:50.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:50.246+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:50.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:50.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:50.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:50.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:50.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:50.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:50.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:50.678+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:51.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:51.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:51.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:51.234+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:51.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:51.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:51.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:51.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:51.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:51.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:51.305+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:51.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:51.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:51.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:51.632+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:52.325 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:52.062+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 2915 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:52.325 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:52.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:52.325 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:52.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:52.325 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:52.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:52.325 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:52.234+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:52.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:52.324+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:52.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:52.324+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:52.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:52.324+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:52.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:52.324+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:52.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:52.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:52.619+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:52.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:52.619+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:53.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:53.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:53.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:53.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:53.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:53.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:53.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:53.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:53.538 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:53.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:53.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:53.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:53.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:53.351+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:53.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:53.352+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:53.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:53.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:53.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:53.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:53.599+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:54.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:54.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:54.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:54.168+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:54.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:54.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:54.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:54.399+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:54.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:54.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:54.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:54.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:54.567+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:55.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:55.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:55.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:55.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:55.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:55.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:55.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:55.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:55.551 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:55.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:55.552 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:55.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:55.552 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:55.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:55.552 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:55.432+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:55.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:55.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:55.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:55.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:55.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:55.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:55.805 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:55.551+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:56.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:56.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:56.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:56.194+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:56.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:56.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:56.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:56.524 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:56.428+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:56.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:56.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:56.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:56.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:56.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:56.522+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:56.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:56.524+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:57.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:33:57.063+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3013 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:33:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:57.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:57.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:57.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:57.229+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:57.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:57.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:57.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:57.401+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:57.484+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:57.484+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:57.484+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:57.485+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:58.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:58.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:58.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:58.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:58.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:58.278+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:58.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:58.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:58.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:58.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:58.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:58.878 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:58.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:58.878 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:58.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:58.878 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:58.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:58.878 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:58.535+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:59.471 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:59.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:59.471 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:59.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:59.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:59.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:33:59.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:33:59.274+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:59.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:59.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:59.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:33:59.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:59.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:59.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:59.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:33:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:33:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:33:59.471+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:00.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:00.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:00.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:00.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:00.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:00.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:00.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:00.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:00.576 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:00.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:00.577 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:00.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:00.577 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:00.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:00.577 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:00.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:00.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:00.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:00.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:00.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:00.575+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:01.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:01.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:01.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:01.269+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:01.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:01.601 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:01.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:01.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:01.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:01.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:01.518+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:01.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:01.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:01.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:01.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:01.600+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:02.264 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:02.064+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3107 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:02.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:02.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:02.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:02.263+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:02.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:02.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:02.641 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:02.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:02.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:02.484+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:02.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:02.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:02.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:02.640+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:03.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:03.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:03.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:03.263+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:03.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:03.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:03.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:03.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:03.540 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:03.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:03.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:03.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:03.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:03.641+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:04.485 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:04.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:04.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:04.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:04.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:04.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:04.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:04.229+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:04.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:04.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:04.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:04.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:04.658+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:04.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:04.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:04.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:04.485+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:05.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:05.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:05.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:05.238+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:05.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:05.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:05.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:05.474+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:05.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:05.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:05.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:05.635+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:05.636+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:06.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:06.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:06.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:06.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:06.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:06.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:06.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:06.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:06.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:06.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:06.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:06.514+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:06.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:06.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:06.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:06.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:06.686+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:07.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:07.066+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3203 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:07.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:07.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:07.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:07.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:07.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:07.235+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:07.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:07.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:07.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:07.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:07.477+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:07.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:07.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:07.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:07.698+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:08.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:08.241+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:08.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:08.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:08.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:08.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:08.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:08.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:08.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:08.468+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:08.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:08.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:08.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:09.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:08.745+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:09.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:09.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:09.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:09.498+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:09.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:09.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:09.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:09.271+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:10.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:09.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:09.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:09.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:09.713+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:10.522 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:10.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:10.522 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:10.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:10.522 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:10.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:10.522 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:10.249+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:10.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:10.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:10.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:10.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:10.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:10.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:10.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:10.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:10.700+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:11.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:11.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:11.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:11.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:11.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:11.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:11.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:11.514+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:11.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:11.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:11.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:11.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:11.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:11.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:11.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:11.297+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:11.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:11.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:11.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:11.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:11.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:11.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:11.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:11.739+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:12.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:12.065+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3297 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:12.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:12.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:12.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:12.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:12.307+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:12.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:12.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:12.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:12.776+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:12.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:12.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:12.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:12.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:13.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:13.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:13.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:13.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:13.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:13.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:13.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:13.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:13.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:13.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:13.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:13.341+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:13.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:13.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:13.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:13.750+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:14.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:14.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:14.520 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:14.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:14.520 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:14.322+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:14.520 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:14.323+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:14.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:14.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:14.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:14.725+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:14.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:14.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:14.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:14.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:15.337+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:15.338+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:15.338+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:15.338+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:15.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:15.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:15.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:15.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:16.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:15.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:16.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:15.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:16.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:15.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:16.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:15.698+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:16.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:16.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:16.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:16.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:16.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:16.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:16.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:16.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:16.347+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:17.068 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:16.720+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:17.068 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:16.720+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:17.068 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:16.720+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:17.068 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:16.720+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:17.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:17.066+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3394 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:17.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:17.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:17.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:17.463+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:17.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:17.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:17.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:17.388+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:18.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:17.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:18.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:17.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:18.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:17.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:18.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:17.765+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:18.696 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:18.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:18.696 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:18.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:18.696 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:18.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:18.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:18.433+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:18.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:18.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:18.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:18.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:18.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:18.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:18.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:18.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:19.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:18.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:19.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:18.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:19.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:18.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:19.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:18.782+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:19.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:19.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:19.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:19.480+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:19.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:19.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:19.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:19.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:19.495+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:20.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:19.796+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:20.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:20.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:20.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:20.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:20.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:20.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:20.528+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:20.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:20.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:20.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:20.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:20.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:20.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:20.797 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:20.481+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:21.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:20.796+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:21.759 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:21.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:21.759 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:21.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:21.759 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:21.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:21.760 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:21.578+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:21.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:21.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:21.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:21.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:21.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:21.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:21.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:21.467+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:22.068 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:21.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:22.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:21.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:22.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:21.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:22.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:21.759+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:22.067+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3486 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:22.744 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:22.459+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:22.744 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:22.459+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:22.744 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:22.459+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:22.744 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:22.459+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:22.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:22.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:22.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:22.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:22.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:22.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:22.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:22.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:23.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:22.743+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:22.743+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:22.743+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:22.743+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:23.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:23.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:23.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:23.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:23.539+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:23.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:23.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:23.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:23.784 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:23.444+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:24.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:23.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:24.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:23.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:24.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:23.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:24.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:23.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:24.807 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:24.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:24.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:24.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:24.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:24.457+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:24.458+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:24.458+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:24.808 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:24.458+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:24.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:24.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:24.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:24.806+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:25.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:25.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:25.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:25.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:25.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:25.503+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:25.777 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:25.504+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:25.777 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:25.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:25.778 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:25.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:25.778 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:25.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:25.778 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:25.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:25.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:25.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:25.775+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:26.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:25.776+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:26.801 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:26.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:26.801 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:26.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:26.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:26.481+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:26.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:26.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:26.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:26.802 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:26.517+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:27.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:26.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:27.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:26.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:27.069 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:26.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:27.070 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:26.801+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:27.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:27.068+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3579 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:27.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:27.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:27.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:27.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:27.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:27.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:27.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:27.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:27.466+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:28.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:27.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:28.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:27.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:28.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:27.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:28.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:27.823+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:28.417+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:28.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:28.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:28.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:28.504+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:28.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:28.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:28.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:28.799+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:29.507 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:29.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:29.507 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:29.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:29.507 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:29.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:29.507 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:29.394+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:29.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:29.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:29.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:29.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:29.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:29.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:29.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:29.507+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:29.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:29.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:29.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:29.793+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:30.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:30.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:30.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:30.396+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:30.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:30.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:30.521+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:30.522+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:30.816+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:30.816+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:30.816+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:30.817+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:31.651 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:31.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:31.651 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:31.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:31.651 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:31.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:31.651 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:31.385+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:31.651 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:31.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:31.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:31.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:31.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:31.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:31.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:31.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:31.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:31.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:31.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:31.863+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:32.338 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:32.068+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3671 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:32.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:32.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:32.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:32.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:32.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:32.337+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:32.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:32.337+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:32.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:32.337+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:32.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:32.338+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:32.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:32.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:32.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:32.821+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:33.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:33.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:33.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:33.335+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:33.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:33.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:33.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:33.501+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:33.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:33.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:33.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:33.805+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:34.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:34.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:34.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:34.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:34.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:34.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:34.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:34.290+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:34.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:34.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:34.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:34.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:34.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:34.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:34.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:34.544+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:34.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:34.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:34.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:35.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:34.851+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:35.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:35.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:35.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:35.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:35.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:35.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:35.579 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:35.316+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:35.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:35.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:35.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:35.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:35.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:35.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:35.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:35.579+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:35.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:35.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:35.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:36.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:35.873+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:36.573 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:36.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:36.573 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:36.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:36.573 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:36.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:36.573 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:36.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:36.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:36.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:36.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:36.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:36.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:36.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:36.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:36.572+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:37.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:37.069+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3767 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:37.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:36.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:37.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:36.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:37.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:36.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:37.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:36.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:37.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:37.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:37.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:37.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:37.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:37.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:37.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:37.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:37.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:37.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:37.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:37.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:37.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:37.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:37.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:37.555+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:38.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:37.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:38.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:37.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:38.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:37.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:38.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:37.894+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:38.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:38.277+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:38.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:39.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:38.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:39.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:38.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:39.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:38.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:39.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:38.866+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:39.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:39.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:39.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:39.305+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:39.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:39.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:39.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:39.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:40.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:39.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:40.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:39.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:40.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:39.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:40.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:39.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:40.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:40.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:40.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:40.315+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:40.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:40.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:40.560+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:40.560+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:40.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:40.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:40.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:40.855+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:41.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:41.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:41.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:41.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:41.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:41.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:41.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:41.347+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:41.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:41.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:41.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:41.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:41.536+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:41.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:41.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:41.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:41.901+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:42.399 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:42.070+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3860 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:42.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:42.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:42.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:42.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:42.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:42.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:42.395+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:42.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:42.396+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:42.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:42.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:42.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:42.913+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:43.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:43.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:43.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:43.356+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:43.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:43.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:43.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:43.590+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:43.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:43.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:43.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:43.927+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:44.631 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:44.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:44.631 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:44.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:44.631 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:44.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:44.631 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:44.332+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:44.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:44.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:44.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:44.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:44.630+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:45.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:44.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:45.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:44.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:45.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:44.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:45.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:44.940+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:45.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:45.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:45.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:45.376+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:45.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:45.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:45.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:45.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:45.621+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:46.424 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:45.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:45.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:45.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:45.962+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:46.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:46.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:46.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:46.424+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:46.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:46.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:46.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:46.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:46.588+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:47.195 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:47.071+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 3955 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:47.195 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:47.195 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:47.195 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:47.195 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:46.926+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:47.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:47.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:47.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:47.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:47.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:47.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:47.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:47.403+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:47.548+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:48.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:47.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:48.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:47.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:48.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:47.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:48.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:47.939+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:48.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:48.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:48.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:48.417+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:48.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:48.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:48.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:48.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:48.595+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:48.946 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:48.944+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:48.946 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:48.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:48.946 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:48.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:49.421 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:48.946+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:49.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:49.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:49.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:49.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:49.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:49.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:49.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:49.422+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:49.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:49.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:49.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:49.618+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:49.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:49.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:49.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:49.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:50.385+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:50.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:50.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:50.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:50.578+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:51.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:50.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:51.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:50.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:51.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:50.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:51.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:50.937+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:51.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:51.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:51.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:51.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:51.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:51.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:51.415+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:51.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:51.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:51.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:51.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:51.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:51.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:51.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:51.601+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:52.195 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:52.072+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4048 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:52.196 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:51.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:52.196 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:51.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:52.196 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:51.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:52.196 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:51.940+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:52.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:52.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:52.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:52.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:52.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:52.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:52.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:52.393+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:52.913 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:52.913 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:52.913 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:52.913 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:52.596+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:52.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:52.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:52.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:52.912+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:53.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:53.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:53.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:53.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:53.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:53.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:53.622 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:53.352+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:53.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:53.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:53.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:53.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:53.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:53.620+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:53.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:53.621+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:54.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:53.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:53.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:53.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:53.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:54.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:54.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:54.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:54.308+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:54.644+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:54.644+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:54.644+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:54.646+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:54.892+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:55.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:55.344+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:55.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:55.344+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:55.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:55.344+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:55.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:55.345+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:55.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:55.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:55.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:55.654+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:56.378 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:55.933+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:56.378 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:55.933+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:56.378 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:55.933+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:56.378 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:55.933+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:56.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:56.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:56.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:56.377+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:56.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:56.699+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:56.934+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:56.934+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:56.934+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:57.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:56.935+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:57.342 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:34:57.072+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4145 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:34:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:57.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:57.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:57.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:57.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:57.957 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:57.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.957 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:57.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.957 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:57.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:57.957 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:57.690+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:58.314 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:57.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:58.314 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:57.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:58.314 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:57.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:58.314 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:57.957+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:58.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:58.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:58.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:58.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:58.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:58.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:58.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:58.314+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:58.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:58.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:58.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:34:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:58.648+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:59.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:58.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:59.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:58.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:59.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:58.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:34:59.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:34:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:34:58.980+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:34:59.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:59.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:59.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:34:59.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:34:59.326+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:00.017 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:59.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.017 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:59.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.017 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:59.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.017 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:34:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:34:59.676+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:00.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:00.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:00.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:00.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:00.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:00.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:00.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:00.017+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:00.664 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:00.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:00.664 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:00.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:00.664 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:00.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:00.664 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:00.300+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:00.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:00.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:00.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:00.663+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:01.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:01.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:01.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:01.067+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:01.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:01.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:01.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:01.312+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:01.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:01.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:01.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:01.642+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:02.073+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4238 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:02.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:02.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:02.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:02.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:02.112+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:02.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:02.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:02.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:02.298+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:02.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:02.649+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:02.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:02.649+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:02.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:02.649+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:02.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:02.650+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:03.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:03.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:03.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:03.155+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:03.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:03.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:03.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:03.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:03.329+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:03.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:03.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:03.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:03.618+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:04.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:04.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:04.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:04.108+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:04.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:04.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:04.346+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:04.347+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:04.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:04.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:04.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:04.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:04.655+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:05.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:05.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:05.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:05.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:05.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:05.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:05.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:05.098+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:05.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:05.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:05.637 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:05.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:05.638 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:05.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:05.638 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:05.360+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:05.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:05.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:05.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:05.637+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:06.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:06.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:06.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:06.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:06.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:06.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:06.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:06.052+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:06.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:06.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:06.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:06.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:06.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:06.314+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:06.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:06.315+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:06.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:06.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:06.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:06.588+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:07.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:07.074+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4333 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:07.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:07.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:07.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:07.082+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:07.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:07.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:07.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:07.313+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:07.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:07.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:07.611+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:07.612+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:08.325 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.041+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:08.325 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.041+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:08.325 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.041+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:08.325 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.042+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:08.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:08.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:08.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:08.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:08.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:08.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:08.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:08.325+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:08.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:08.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:08.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:08.597+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:09.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:09.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:09.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:09.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:08.994+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:09.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:09.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:09.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:09.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:09.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:09.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:09.334+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:09.335+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:10.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:10.011+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:10.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:10.011+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:10.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:10.011+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:10.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:10.012+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:10.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:10.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:10.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:10.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:10.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:10.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:10.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:10.298+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:10.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:10.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:10.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:10.589+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:11.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.032+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:11.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.032+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:11.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.032+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:11.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.033+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:11.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:11.305+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:11.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:11.305+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:11.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:11.305+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:11.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:11.306+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:11.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:11.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:11.563+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:11.564+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:12.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:12.075+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4429 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:12.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.991+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:12.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.991+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:12.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.991+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:12.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:11.991+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:12.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:12.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:12.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:12.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:12.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:12.297+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:12.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:12.298+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:12.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:12.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:12.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:12.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:12.583+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:13.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:13.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:13.316 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:13.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:13.317 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:13.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:13.317 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:13.035+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:13.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:13.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:13.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:13.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:13.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:13.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:13.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:13.316+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:13.568 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:13.568 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:13.568 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:13.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:13.568 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:13.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:14.338 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:14.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:14.338 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:14.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:14.338 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:14.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:14.338 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:14.078+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:14.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:14.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:14.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:14.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:14.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:14.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:14.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:14.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:14.337+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:15.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:15.065+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:15.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:15.065+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:15.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:15.065+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:15.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:15.066+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:15.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:15.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:15.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:15.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:15.352+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:15.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:15.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:15.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:15.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:16.394 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:16.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:16.394 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:16.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:16.394 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:16.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:16.395 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:16.093+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:16.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:16.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:16.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:16.531+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:16.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:16.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:16.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:16.393+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:17.390 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:17.075+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4525 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:17.391 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:17.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:17.391 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:17.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:17.391 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:17.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:17.391 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:17.116+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:17.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:17.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:17.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:17.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:17.390+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:17.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:17.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:17.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:17.549+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:18.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:18.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:18.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:18.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:18.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:18.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:18.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:18.148+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:18.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:18.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:18.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:18.540+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:18.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:18.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:18.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:18.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:18.402+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:19.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:19.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:19.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:19.412+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:19.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:19.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:19.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:19.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:19.180+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:19.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:19.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:19.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:19.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:19.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:19.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:19.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:19.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:20.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:20.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:20.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:20.151+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:20.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:20.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:20.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:20.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:20.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:20.443+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:20.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:20.444+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:20.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:20.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:20.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:20.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:21.532 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:21.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:21.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:21.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:21.400+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:21.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:21.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:21.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:21.533 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:21.195+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:21.821 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:21.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:21.821 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:21.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:21.821 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:21.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:21.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:21.532+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:22.362 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:22.076+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4628 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:22.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:22.171+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:22.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:22.171+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:22.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:22.171+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:22.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:22.172+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:22.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:22.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:22.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:22.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:22.361+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:22.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:22.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:22.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:22.537+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:23.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:23.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:23.336+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:23.338+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:23.143+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:23.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:23.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:23.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:23.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:23.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:24.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:24.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:24.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:24.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:24.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:24.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:24.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:24.102+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:24.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:24.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:24.374+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:24.375+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:24.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:24.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:24.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:24.507+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:25.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:25.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:25.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:25.094+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:25.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:25.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:25.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:25.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:25.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:25.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:25.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:25.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:25.384+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:26.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:26.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:26.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:26.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:26.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:26.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:26.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:26.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:26.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:26.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:26.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:26.500+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:26.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:26.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:26.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:26.405+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:27.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:27.077+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4728 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:27.419 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:27.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:27.419 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:27.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:27.419 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:27.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:27.419 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:27.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:27.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:27.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:27.483+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:27.484+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:27.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:27.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:27.417+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:27.418+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:28.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:28.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:28.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:28.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:28.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:28.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:28.418 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:28.049+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:28.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:28.416+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:28.417+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:28.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:28.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:28.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:28.453+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:29.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:29.029+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:29.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:29.029+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:29.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:29.029+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:29.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:29.029+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:29.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:29.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:29.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:29.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:29.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:29.442+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:29.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:29.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:29.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:29.477+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:30.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:30.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:30.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:30.017+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:30.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:30.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:30.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:30.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:30.440+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:30.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:30.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:31.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:31.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:31.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:31.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:31.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:31.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:31.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:31.068+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:31.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:31.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:31.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:31.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:31.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:31.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:31.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:31.393+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:31.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:31.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:31.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:31.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:31.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:31.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:31.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:31.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:32.377 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:32.078+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4822 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:32.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:32.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:32.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:32.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:32.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:32.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:32.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:32.071+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:32.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:32.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:32.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:32.375+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:32.376+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:32.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:32.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:32.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:32.516+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:33.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:33.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:33.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:33.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:33.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:33.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:33.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:33.068+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:33.410+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:33.410+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:33.410+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:33.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:33.411+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:33.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:33.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:33.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:33.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:33.493+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:34.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:34.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:34.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:34.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:34.096+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:34.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:34.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:34.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:34.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:34.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:34.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:34.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:34.509+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:34.440+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:34.440+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:34.440+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:34.441+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:35.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:35.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:35.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:35.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:35.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:35.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:35.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:35.053+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:35.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:35.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:35.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:35.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:35.438+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:35.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:35.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:35.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:35.505+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:36.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:36.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:36.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:36.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.026+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:36.449+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:36.449+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:36.449+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:36.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:36.450+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:36.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:36.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:36.472+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:36.472+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:37.078+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 4920 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:36.982+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:37.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:37.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:37.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:37.426+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:37.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:37.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:37.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:37.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:38.385 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:37.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:38.385 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:37.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:38.385 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:37.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:38.385 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:37.942+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:38.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:38.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:38.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:38.384+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:38.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:38.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:38.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:38.407+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:39.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:38.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:39.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:38.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:39.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:38.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:39.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:38.981+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:39.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:39.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:39.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:39.392+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:39.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:39.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:39.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:39.407+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:40.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:39.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:40.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:39.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:40.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:39.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:40.414 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:39.936+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:40.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:40.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:40.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:40.414+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:40.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:40.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:40.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:40.440+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:41.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:40.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:41.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:40.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:41.412 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:40.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:41.412 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:40.975+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:41.457+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:41.457+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:41.457+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:41.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:41.458+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:41.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:41.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:41.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:41.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:42.079+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5030 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:41.955+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:41.955+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:42.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:41.955+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:41.956+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:42.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:42.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:42.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:42.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:42.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:42.463+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:42.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:42.464+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:42.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:42.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:42.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:42.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:43.325 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:42.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:42.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:42.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:43.327 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:42.943+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:43.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:43.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:43.497+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:43.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:43.497+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:43.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:43.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:43.490+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:43.490+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:43.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:43.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:43.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:44.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:43.905+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:44.456+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:44.456+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:44.457+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:44.457+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:44.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:44.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:44.447+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:44.448+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:44.935+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:44.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:44.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:44.936+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:45.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:45.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:45.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:45.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:45.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:45.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:45.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:45.436+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:45.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:45.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:45.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:45.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:45.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:45.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:45.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:45.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:46.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:45.923+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:46.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:46.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:46.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:46.403+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:46.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:46.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:46.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:46.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:47.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:47.080+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5125 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:47.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:46.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:47.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:46.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:47.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:46.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:47.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:46.964+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:47.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:47.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:47.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:47.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:47.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:47.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:47.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:47.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:47.388+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:47.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:47.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:47.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:48.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:47.927+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:48.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:48.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:48.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:48.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:48.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:48.432+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:48.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:48.424+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:48.424+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:48.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:48.424+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:48.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:48.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:48.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:48.917+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:49.380+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:49.380+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:49.380+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:49.381+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:49.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:49.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:49.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:49.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:49.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:49.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:49.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:50.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:49.881+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:50.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:50.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:50.385+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:50.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:50.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:50.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:50.405+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:50.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:50.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:50.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:50.921+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:51.656 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:51.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:51.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:51.384+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:51.384+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:51.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:51.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:51.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:51.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:51.391+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:51.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:51.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:51.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:51.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:51.878+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:52.342 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:52.081+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5220 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:52.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:52.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:52.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:52.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:52.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:52.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:52.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:52.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:52.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:52.400+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:52.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:52.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:52.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:53.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:52.910+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:53.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:53.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:53.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:53.294+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:53.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:53.548 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:53.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:53.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:53.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:53.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:53.405+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:53.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:53.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:53.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:53.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:54.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:54.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:54.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:54.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:54.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:54.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:54.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:54.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:54.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:54.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:54.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:54.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:54.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:54.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:54.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:54.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:55.613 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:55.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:55.613 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:55.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:55.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:55.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:55.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:55.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:55.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:55.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:55.378+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:56.253 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:55.937+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:56.253 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:55.937+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:56.253 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:55.937+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:56.253 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:55.938+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:56.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:56.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:56.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:56.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:56.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:56.253+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:56.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:56.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:56.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:56.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:57.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:35:57.081+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5324 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:35:57.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:56.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:56.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:56.909+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:56.910+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:57.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:57.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:57.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:57.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:57.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:57.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:57.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:57.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:57.435+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:58.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:57.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:58.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:57.912+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:58.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:57.912+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:58.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:57.913+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:58.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:58.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:58.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:58.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:58.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:58.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:58.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:58.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:58.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:59.149 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:58.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:59.149 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:58.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:59.149 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:58.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:35:59.149 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:58.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:59.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:59.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:59.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:35:59.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:59.386+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:59.386+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:59.386+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:35:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:35:59.387+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:00.133 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:59.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:00.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:59.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:00.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:59.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:00.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:35:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:35:59.880+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:00.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:00.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:00.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:00.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:00.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:00.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:00.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:00.377+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:01.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:00.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:01.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:00.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:00.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:00.915+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:01.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:01.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:01.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:01.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:01.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:01.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:01.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:01.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:01.657 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:01.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:01.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:01.337+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:02.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:02.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:02.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:02.121+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:01.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:01.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:01.941+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:02.208 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:01.941+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:02.209 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:02.082+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5433 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:02.347+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:02.347+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:02.347+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:02.349+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:03.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:03.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:03.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:03.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:03.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:02.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:02.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:02.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:03.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:02.978+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:03.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:03.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:03.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:03.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:03.310+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:04.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:04.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:04.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:04.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:03.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:03.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:03.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:04.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:03.950+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:04.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:04.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:04.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:04.342+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:05.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:04.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:05.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:04.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:05.098 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:04.939+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:05.098 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:04.939+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:05.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:05.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:05.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:05.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:05.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:05.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:05.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:05.295+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:06.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:06.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:06.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:06.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:06.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:06.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:06.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:06.119+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:06.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:05.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:06.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:05.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:06.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:05.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:06.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:05.951+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:06.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:06.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:06.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:06.267+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:07.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:07.083+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5544 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:07.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:06.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:06.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:06.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:06.909+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:07.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:07.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:07.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:07.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:07.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:07.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:07.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:07.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:07.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:08.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:08.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:08.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:08.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:07.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:07.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:07.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:07.951+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:08.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:08.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:08.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:08.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:09.301 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:09.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:09.301 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:09.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:09.301 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:09.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:09.301 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:09.111+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:09.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:08.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:09.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:08.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:09.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:08.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:09.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:08.961+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:09.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:09.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:09.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:09.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:09.301+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:10.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:10.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:10.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:10.102+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:09.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:09.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:09.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:10.267 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:09.929+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:10.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:10.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:10.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:10.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:10.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:11.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:11.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:11.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:11.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:11.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:10.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:10.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:10.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:11.250 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:10.930+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:11.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:11.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:11.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:11.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:11.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:11.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:11.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:11.249+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:12.211 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:12.084+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5636 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:12.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:12.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:12.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:12.071+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:11.925+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:11.925+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:12.212 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:11.925+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:12.213 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:11.925+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:12.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:12.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:12.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:12.239+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.026+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:12.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:12.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:12.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:12.878+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:13.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:13.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:13.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:13.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:13.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:13.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:13.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:13.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:13.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:13.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:13.920+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:14.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:13.921+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:13.984+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:14.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:14.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:14.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:14.252+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:15.281 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.000+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:15.281 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.000+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:15.281 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:15.281 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.001+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:15.282 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:14.967+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:15.282 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:14.967+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:15.282 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:14.967+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:15.282 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:14.967+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:15.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:15.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:15.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:15.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:15.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:16.300 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:16.300 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:16.300 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:16.300 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:15.952+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:16.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:15.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:16.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:15.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:16.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:15.950+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:16.301 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:15.950+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:16.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:16.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:16.298+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:16.299+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:17.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:17.085+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5729 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:17.295 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:16.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:17.295 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:16.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:16.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:16.960+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:16.971+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:16.971+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:16.971+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:17.296 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:16.971+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:17.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:17.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:17.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:17.296+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:17.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:17.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:17.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:17.998+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:17.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:17.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:17.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.309 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:17.928+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:18.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:18.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:18.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:18.309+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:18.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:18.881+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:18.881+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:18.881+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:18.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:18.881+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:19.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:18.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:19.297 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:18.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:19.297 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:18.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:19.297 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:18.982+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:19.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:19.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:19.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:19.297+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:20.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:19.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:20.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:19.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:20.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:19.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:20.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:19.979+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:19.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:19.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:19.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:19.923+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:20.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:20.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:20.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:20.267+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:21.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:20.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:21.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:20.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:21.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:20.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:21.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:20.983+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:21.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:20.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:21.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:20.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:21.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:20.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:21.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:20.937+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:21.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:21.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:21.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:21.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:21.659 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:21.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:21.659 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:21.278+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:22.304 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:22.086+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5822 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:22.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:22.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:22.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:21.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:22.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:21.996+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:22.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:21.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:22.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:21.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:22.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:21.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:22.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:21.972+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:22.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:22.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:22.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:22.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:22.304+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:23.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:22.975+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:23.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:22.975+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:23.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:22.975+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:23.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:22.975+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:23.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:22.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:23.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:22.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:23.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:22.929+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:23.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:22.930+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:23.550 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:23.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:23.550 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:23.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:23.550 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:23.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:23.550 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:23.279+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:23.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:23.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:23.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:23.998+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:23.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:23.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:23.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:24.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:23.972+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:24.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:24.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:24.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:24.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:25.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:24.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:25.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:24.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:25.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:24.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:25.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:24.980+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:25.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:24.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:25.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:24.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:25.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:24.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:25.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:24.988+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:25.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:25.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:25.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:25.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:26.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:26.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:26.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:26.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:26.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:26.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:26.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:26.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:26.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:26.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:26.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:26.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:26.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:26.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:26.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:26.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:26.024+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:27.303 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:27.087+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 5915 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:27.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:27.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:27.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:27.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:27.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:27.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:27.304 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:27.055+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:27.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:27.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:27.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:27.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:27.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:27.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:27.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:27.046+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:27.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:27.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:27.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:27.302+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:28.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:28.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:28.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:28.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:28.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:28.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:28.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:28.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:28.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:28.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:28.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:28.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:28.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:28.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:28.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:28.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:28.342+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:28.342+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:28.342+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:28.342+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:29.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:29.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:29.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:29.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:29.054+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:29.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:29.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:29.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:29.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:29.085+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:29.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:29.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:29.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:29.335+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:30.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:30.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:30.331 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:30.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:30.025+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:30.026+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:30.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:30.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:30.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:30.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:30.107+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:30.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:30.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:30.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:30.330+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:31.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:31.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:31.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:31.028+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:31.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:31.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:31.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:31.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:31.138+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:31.658 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:31.659 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:31.659 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:31.659 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:31.304+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:32.349 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:32.088+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6008 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:32.349 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:32.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:32.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:32.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:32.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:32.090+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:32.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:32.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:32.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:32.092+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:32.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:32.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:32.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:32.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:32.348+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:33.319 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:33.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:33.319 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:33.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:33.319 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:33.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:33.320 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:33.052+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:33.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:33.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:33.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:33.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:33.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:33.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:33.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:33.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:33.318+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:33.318+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:33.318+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:33.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:33.318+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:34.276 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:34.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:34.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:34.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:34.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:34.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:34.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:34.096+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:34.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:34.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:34.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:34.276+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:35.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:35.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:35.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:35.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:35.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:35.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:35.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:35.260+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:35.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:35.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:35.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:35.136+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:36.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:36.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:36.054+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:36.055+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:36.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:36.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:36.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:36.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:36.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:36.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:36.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:36.131+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:37.089+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6103 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:37.129+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:37.129+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:37.129+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:37.130+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:37.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:37.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:37.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:37.245+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:37.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:37.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:37.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:37.086+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:38.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:38.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:38.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:38.289+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:38.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:38.056+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:38.056+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:38.057+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:38.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:38.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:38.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:38.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:39.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:39.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:39.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:39.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:39.099+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:39.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:39.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:39.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:39.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:39.310+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:39.310+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:39.310+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:39.311+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:40.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:40.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:40.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:40.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:40.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:40.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:40.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:40.045+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:40.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:40.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:40.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:40.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:40.077+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:40.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:40.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:40.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:40.354+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:41.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:41.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:41.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:41.041+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:41.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:41.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:41.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:41.395 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:41.119+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:41.660 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:41.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:41.660 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:41.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:41.660 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:41.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:41.660 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:41.394+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:42.090+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6196 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:42.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:42.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:42.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:42.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:42.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:42.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:42.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:42.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:42.102+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:42.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:42.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:42.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:42.436+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:43.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:43.047+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:43.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:43.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:43.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:43.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:43.056+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:43.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:43.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:43.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:43.405+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:44.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:44.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:44.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:44.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:44.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:44.040+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:44.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:44.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:44.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:44.030+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:44.765 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:44.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:44.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:44.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:44.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:44.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:44.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:44.453+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:45.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:45.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:45.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:45.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:45.007+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:45.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:45.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:45.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:45.007+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:45.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:45.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:45.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:45.447+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.019+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:46.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:46.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:46.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:46.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:46.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:47.091+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6289 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:47.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:47.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:47.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:47.047+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:46.996+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:47.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:47.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:47.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:47.477+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:48.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:48.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:48.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:48.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:48.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:48.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:48.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:48.045+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:48.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:48.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:48.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:48.500+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:49.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:49.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:49.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:49.091+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:49.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:49.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:49.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:49.053+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:49.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:49.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:49.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:49.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:50.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:50.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:50.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:50.091+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:50.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:50.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:50.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:50.080+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:50.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:50.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:50.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:50.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:51.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:51.475 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:51.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:51.475 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:51.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:51.475 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:51.049+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:51.475 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:51.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:51.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:51.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:51.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:51.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:51.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:51.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:51.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:51.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:51.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:51.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:52.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:52.092+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6382 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:52.024+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:52.024+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:52.024+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:52.024+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:52.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:52.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:52.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:52.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:52.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:52.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:52.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:52.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:53.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:53.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:53.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:53.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:53.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:53.069+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:53.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:53.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:53.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:53.099+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:53.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:53.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:53.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:53.477+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:54.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:54.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:54.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:54.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:54.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:54.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:54.093+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:54.093+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:54.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:54.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:54.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:54.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:54.451+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:55.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:55.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:55.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:55.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:55.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:55.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:55.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:55.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:55.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:55.472+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:56.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:56.075+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:56.075+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:56.075+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:56.076+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:55.994+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:56.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:56.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:56.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:56.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:56.495+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:57.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:36:57.093+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6477 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:36:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:57.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:57.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:57.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:57.120+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:56.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:56.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:56.982+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:56.983+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:57.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:57.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:57.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:57.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:58.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:57.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:57.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:57.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:57.987+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:58.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:58.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:58.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:58.170+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:58.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:58.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:58.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:58.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:58.469+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:58.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:58.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:58.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:58.965+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:59.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:59.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:59.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:36:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:36:59.202+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:36:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:59.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:59.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:59.504+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:36:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:36:59.505+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:00.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:59.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:00.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:59.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:00.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:59.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:00.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:36:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:36:59.962+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:00.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:00.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:00.552 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:00.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:00.552 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:00.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:00.552 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:00.237+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:00.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:00.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:00.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:00.549+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:01.245 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:00.960+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:01.245 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:00.960+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:01.245 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:00.960+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:01.245 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:00.960+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:01.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:01.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:01.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:01.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:01.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:01.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:01.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:01.244+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:01.914 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:01.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:01.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:01.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:01.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:01.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:01.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:01.526+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:02.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:02.094+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6570 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:02.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:01.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:02.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:01.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:01.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:01.915+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:02.526 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:02.272+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:02.526 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:02.272+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:02.526 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:02.272+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:02.526 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:02.272+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:02.899 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:02.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:02.900 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:02.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:02.900 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:02.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:02.900 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:02.526+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:02.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:02.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:03.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:02.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:03.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:02.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:03.510 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:03.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:03.510 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:03.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:03.510 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:03.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:03.510 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:03.293+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:03.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:03.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:03.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:03.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:03.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:04.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:03.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:04.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:03.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:04.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:03.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:04.261 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:03.928+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:04.520 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:04.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:04.520 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:04.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:04.520 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:04.261+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:04.520 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:04.262+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:04.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:04.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:04.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:04.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:04.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:04.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:04.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:04.520+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:04.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:04.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:04.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:04.916+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:05.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:05.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:05.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:05.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:05.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:05.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:05.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:05.252+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:05.550+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:05.550+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:05.550+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:05.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:05.551+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:06.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:05.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:06.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:05.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:06.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:05.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:06.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:05.933+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:06.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:06.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:06.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:06.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:06.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:06.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:06.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:06.238+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:06.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:06.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:06.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:06.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:06.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:07.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:07.094+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6664 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:06.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:06.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:06.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:06.965+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:07.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:07.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:07.222 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:07.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:07.223 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:07.212+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:07.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.923698+0000 front 2024-01-19T06:26:35.923761+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:07.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.923822+0000 front 2024-01-19T06:26:35.923734+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:07.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.923741+0000 front 2024-01-19T06:26:35.923798+0000 (oldest deadline 2024-01-19T06:27:01.823646+0000) 2024-01-19T06:37:07.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:07.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:07.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:07.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:07.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:07.967+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:08.169+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:08.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:08.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:09.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:09.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:09.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:09.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:09.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:09.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:09.342 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:09.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:09.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:09.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:09.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:09.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:09.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:09.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:09.343 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:09.134+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:09.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:09.447+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:10.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:10.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:10.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:10.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:10.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:10.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:10.107+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:10.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:10.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:10.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:10.022+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:10.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:10.469+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:11.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:11.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:11.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:11.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:11.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:11.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:11.053+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:11.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:11.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:11.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:11.072+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:11.449+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:12.405 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:12.095+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6758 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:12.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:12.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:12.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:12.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:12.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:12.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:12.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:12.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:12.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:12.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:12.113+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:12.404+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:13.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:13.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:13.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:13.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:13.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:13.045+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:13.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:13.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:13.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:13.413 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:13.102+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:13.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:13.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:14.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:14.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:14.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:14.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:14.080+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:14.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:14.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:14.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:14.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:14.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:14.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:15.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:15.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:15.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:15.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:15.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:15.103+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:15.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:15.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:15.040+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:15.041+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:15.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:16.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:16.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:16.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:16.088+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:16.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:16.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:16.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:16.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:16.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:16.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:17.395 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:17.096+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6853 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:17.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:17.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:17.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:17.108+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:17.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:35.752313+0000 front 2024-01-19T06:26:35.752244+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:17.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:35.752261+0000 front 2024-01-19T06:26:35.752280+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:17.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:35.752338+0000 front 2024-01-19T06:26:35.752355+0000 (oldest deadline 2024-01-19T06:27:00.452247+0000) 2024-01-19T06:37:17.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:17.167+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:17.394+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:18.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:18.120+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:18.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:18.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:18.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:18.087+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:18.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:18.437+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:19.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:19.138+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:19.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:19.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:19.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:19.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:19.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:19.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:20.122+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:20.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:20.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 since back 2024-01-19T06:26:38.140035+0000 front 2024-01-19T06:26:38.140155+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:20.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 since back 2024-01-19T06:26:38.140117+0000 front 2024-01-19T06:26:38.140173+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:20.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 since back 2024-01-19T06:26:38.140193+0000 front 2024-01-19T06:26:38.140141+0000 (oldest deadline 2024-01-19T06:27:03.440036+0000) 2024-01-19T06:37:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:20.121+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:20.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:20.441+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:21.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:37:21.184+0000 7f18d9198700 0 monclient(hunting): authenticate timed out after 300 2024-01-19T06:37:21.186 INFO:teuthology.orchestra.run.smithi134.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-19T06:37:21.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:21.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:21.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:21.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:21.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:21.394+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:22.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-01-19T06:37:22.381 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/mon.smithi134/config 2024-01-19T06:37:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:22.097+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 6947 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:22.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:22.109+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:22.180+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:22.771 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:22.429+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:23.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:23.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:23.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:23.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:23.420+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:24.396 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:24.136+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:24.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:24.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:24.395+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:25.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:25.144+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:25.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:25.375+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:25.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:26.160+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:26.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:26.409+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:26.161+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:27.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:27.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:27.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:27.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:27.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:27.098+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7041 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:27.774 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:27.442+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:28.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:28.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:28.401 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:28.145+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:28.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:28.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:28.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:28.400+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:29.167+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:29.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:29.161+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:29.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:29.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:29.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:29.440+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:30.198+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:30.159+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:30.426+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:30.426+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:30.426+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:30.427+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:31.663 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:31.219+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:31.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:31.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:31.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:31.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:31.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:31.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:31.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:31.414+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:31.664 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:31.203+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:32.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:32.099+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7135 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:32.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:32.195+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:32.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:32.195+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:32.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:32.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:32.423+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:32.424+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:33.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:33.173+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:33.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:33.178+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:33.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:33.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:33.918 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:33.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:33.918 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:33.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:33.918 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:33.463+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:34.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:34.159+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:34.453 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:34.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:34.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:34.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:34.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:34.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:34.453+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:35.167+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:35.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:35.157+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:35.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:35.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:35.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:35.465+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:36.158+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:36.187+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:36.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:36.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:36.477+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:36.478+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:37.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:37.223+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:37.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:37.100+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7230 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:37.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:37.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:37.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:37.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:37.462+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:38.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:38.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:38.465 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:38.251+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:38.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:38.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:38.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:38.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:38.463+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:39.462 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:39.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:39.462 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:39.255+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:39.914 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:39.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:39.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:39.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:39.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:39.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:39.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:39.462+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:40.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:40.160+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:40.544 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:40.293+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:40.545 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:40.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:40.545 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:40.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:40.545 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:40.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:40.545 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:40.500+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:41.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:41.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:41.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:41.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:41.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:41.274+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:41.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:41.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:41.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:41.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:41.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:42.101+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7324 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:42.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:42.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:42.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:42.154+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:42.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:42.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:42.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:42.324+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:42.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:42.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:42.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:42.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:43.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:43.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:43.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:43.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:43.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:43.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:43.355+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:43.355+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:43.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:43.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:43.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:43.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:44.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:44.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:44.331+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:44.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:44.332+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:44.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:44.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:44.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:44.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:44.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:44.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:44.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:44.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:44.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:44.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:44.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:44.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:44.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:44.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:44.417+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:45.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:45.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:45.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:45.103+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:45.307+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:45.307+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:45.307+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:45.307+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:45.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:45.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:45.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:45.458+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:46.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:46.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:46.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:46.126+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:46.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:46.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:46.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:46.289+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:46.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:46.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:47.103+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7418 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:47.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:47.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:47.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:47.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:47.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:47.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:47.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:47.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:47.267+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:47.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:47.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:47.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:47.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:48.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:48.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:48.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:48.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:48.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:48.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:48.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:48.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:48.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:48.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:48.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:48.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:49.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:49.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:49.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:49.303+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:49.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:49.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:49.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:49.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:49.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:49.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:49.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:49.552+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:50.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:50.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:50.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:50.299+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:50.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:50.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:50.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:50.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:50.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:50.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:50.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:50.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:51.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:51.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:51.254+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:51.255+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:51.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:51.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:51.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:51.068+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:51.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:51.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:51.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:51.524+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:52.349 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:52.103+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7512 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:52.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:52.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:52.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:52.065+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:52.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:52.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:52.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:52.350 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:52.216+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:52.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:52.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:52.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:52.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:52.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:52.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:52.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:52.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:53.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:53.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:53.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:53.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:53.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:53.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:53.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:53.214+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:53.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:53.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:53.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:53.504+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:54.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:54.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:54.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:54.239+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:54.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:54.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:54.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:54.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:55.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:55.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:54.979+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:55.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:55.199+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:55.199+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:55.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:55.199+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:55.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:55.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:55.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:55.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:55.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:56.197 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:55.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:56.197 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:55.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:56.197 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:55.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:56.197 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:55.942+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:56.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:56.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:56.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:56.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:56.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:56.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:56.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:56.196+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:56.906 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:56.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:56.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:56.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:56.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:56.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:56.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:56.526+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:57.165 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:37:57.104+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7608 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:37:57.165 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:56.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:57.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:56.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:57.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:56.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:57.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:56.906+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:57.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:57.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:57.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:57.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:57.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:57.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:57.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:57.904 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:57.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:57.904 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:57.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:57.904 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:57.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:57.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:57.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:57.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:57.902+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:58.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:58.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:58.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:58.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:58.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:58.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:58.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:58.122+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:58.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:58.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:58.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:58.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:58.889 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:58.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:58.889 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:58.540+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:59.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:59.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:59.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:59.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:59.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:59.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:37:59.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:37:59.126+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:58.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:58.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:58.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:37:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:58.888+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:37:59.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:59.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:59.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:59.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:59.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:59.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:37:59.905 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:37:59.545+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:00.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:59.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:00.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:59.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:00.172 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:59.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:00.172 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:37:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:37:59.904+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:00.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:00.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:00.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:00.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:00.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:00.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:00.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:00.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:00.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:00.531+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:00.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:00.532+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:00.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:00.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:00.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:00.860+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:01.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:01.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:01.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:01.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:01.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:01.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:01.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:01.179+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:01.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:01.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:01.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:01.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:01.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:01.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:01.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:01.542+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:02.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:01.839+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:02.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:01.839+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:02.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:01.839+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:02.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:01.839+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:02.105+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7701 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:02.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:02.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:02.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:02.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:02.189+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:02.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:02.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:02.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:02.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:02.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:02.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:02.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:02.549+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:02.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:02.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:03.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:02.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:03.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:02.876+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:03.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:03.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:03.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:03.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:03.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:03.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:03.557 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:03.188+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:03.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:03.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:03.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:03.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:03.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:03.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:03.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:03.560+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:04.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:03.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:04.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:03.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:04.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:03.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:04.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:03.861+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:04.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:04.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:04.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:04.155+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:04.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:04.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:04.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:04.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:04.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:04.537+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:04.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:04.538+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:04.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:04.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:04.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:05.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:04.875+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:05.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:05.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:05.512 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:05.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:05.175+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:05.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:05.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:05.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:05.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:05.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:05.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:05.915 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:05.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:05.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:05.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:05.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:05.914+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:06.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:06.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:06.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:06.153+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:06.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:06.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:06.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:06.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:06.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:06.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:06.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:06.485+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:07.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:07.106+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7796 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:07.132 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:06.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:07.132 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:06.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:07.132 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:06.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:07.132 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:06.879+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:07.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:07.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:07.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:07.167+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:07.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:07.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:07.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:07.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:07.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:07.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:07.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:07.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:08.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:08.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:08.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:08.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:08.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:08.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:08.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:08.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:07.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:07.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:07.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:07.891+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:08.908 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:08.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:08.909 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:08.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:08.909 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:08.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:08.909 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:08.460+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:08.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:08.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:08.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:09.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:08.908+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:09.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:09.176+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:09.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:09.176+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:09.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:09.176+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:09.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:09.176+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:09.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:09.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:09.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:09.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:09.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:09.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:09.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:09.466+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:09.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:09.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:09.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:09.888+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:10.470 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:10.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:10.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:10.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:10.419+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:10.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:10.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:10.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:10.471 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:10.206+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:10.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:10.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:10.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:10.910+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:11.665 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:11.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:11.665 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:11.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:11.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:11.449+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:11.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:11.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:11.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:11.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:11.228+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:12.233 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:12.107+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7889 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:12.233 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:11.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:12.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:11.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:12.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:11.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:12.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:11.948+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:12.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:12.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:12.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:12.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:12.402+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:12.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:12.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:12.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:12.257+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:13.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:12.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:13.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:12.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:13.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:12.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:13.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:12.966+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:13.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:13.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:13.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:13.446+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:13.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:13.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:13.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:13.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:13.283+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:13.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:13.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:13.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:13.946+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:14.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:14.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:14.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:14.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:14.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:14.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:14.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:14.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:14.276+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:14.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:14.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:14.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:14.904+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:15.499 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:15.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:15.499 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:15.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:15.500 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:15.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:15.500 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:15.248+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:15.923 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:15.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:15.923 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:15.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:15.923 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:15.498+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:15.923 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:15.499+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:15.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:15.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:15.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:15.922+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:16.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:16.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:16.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:16.464+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:16.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:16.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:16.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:16.240+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:17.233 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:17.108+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 7985 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:17.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:16.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:17.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:16.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:17.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:16.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:17.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:16.947+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:17.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:17.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:17.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:17.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:17.469+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:17.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:17.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:17.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:17.288+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:18.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:17.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:18.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:17.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:18.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:17.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:18.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:17.988+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:18.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:18.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:18.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:18.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:18.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:18.297+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:18.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:18.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:18.298+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:19.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:19.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:19.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:18.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:19.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:18.978+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:19.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:19.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:19.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:19.347+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:19.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:19.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:19.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:19.486+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:20.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:19.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:20.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:19.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:20.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:19.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:20.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:19.929+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:20.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:20.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:20.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:20.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:20.449+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:20.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:20.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:20.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:20.309+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:20.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:20.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:20.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:20.884+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:21.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:21.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:21.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:21.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:21.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:21.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:21.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:21.408+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:21.667 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:21.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:21.667 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:21.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:21.667 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:21.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:21.667 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:21.354+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:22.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:22.109+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8078 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:21.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:21.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:21.915+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:21.915+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:22.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:22.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:22.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:22.373+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:22.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:22.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:22.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:22.457+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:23.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:22.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:23.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:22.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:23.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:22.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:23.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:22.963+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:23.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:23.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:23.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:23.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:23.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:23.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:23.328+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:24.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:23.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:24.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:23.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:24.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:23.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:24.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:23.964+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:24.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:24.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:24.496+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:24.497+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:24.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:24.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:24.344+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:24.345+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:25.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:24.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:25.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:24.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:24.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:24.916+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:25.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:25.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:25.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:25.525+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:25.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:25.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:25.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:25.317+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:25.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:25.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:25.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:26.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:25.944+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:26.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:26.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:26.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:26.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:26.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:26.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:26.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:26.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:27.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:27.110+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8172 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:27.236 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:26.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:27.236 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:26.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:27.236 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:26.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:27.236 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:26.959+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:27.606 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:27.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:27.607 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:27.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:27.607 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:27.413+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:27.607 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:27.414+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:27.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:27.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:27.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:27.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:27.606+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:28.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:27.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:28.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:27.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:28.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:27.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:28.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:27.998+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:28.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:28.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:28.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:28.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:28.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:28.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:28.638 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:28.368+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:28.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:28.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:28.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:28.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:28.637+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:29.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:29.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:29.341 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:29.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:29.341 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:29.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:29.341 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:29.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:29.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:29.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:29.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:29.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:29.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:29.339+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:29.595 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:29.340+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:29.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:29.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:29.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:29.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:29.594+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:30.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:30.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:30.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:30.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:30.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:30.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:30.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:30.018+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:30.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:30.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:30.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:30.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:30.634 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:30.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:30.635 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:30.313+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:30.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:30.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:30.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:30.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:30.633+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:31.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:31.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:31.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:31.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:31.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:31.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:31.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:31.294+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:31.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:31.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:31.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:31.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:31.672+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:32.286 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:32.112+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8264 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:32.286 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:32.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:32.286 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:32.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:32.286 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:32.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:32.286 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:32.011+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:32.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:32.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:32.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:32.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:32.664 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:32.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:32.664 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:32.337+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:32.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:32.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:32.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:32.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:32.664+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:33.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:33.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:33.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:33.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:33.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:33.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:33.363 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:33.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:33.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:33.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:33.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:33.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:33.363+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:34.029 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:33.705+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:34.029 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:33.705+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:34.029 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:33.705+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:34.029 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:33.705+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:34.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:34.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:34.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:34.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:34.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:34.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:34.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:34.029+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:34.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:34.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:34.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:34.395+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:35.032 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:34.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.032 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:34.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.032 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:34.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.032 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:34.685+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:35.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:35.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:35.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:35.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:35.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:35.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:35.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:35.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:35.657 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:35.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:35.658 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:35.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:35.658 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:35.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:35.658 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:35.405+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:35.656+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:35.656+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:35.656+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:35.657+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:36.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:36.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:36.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:36.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:36.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:36.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:36.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:36.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:36.434+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:36.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:36.647+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:37.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:37.112+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8361 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:37.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:37.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:37.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:37.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:37.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:37.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:37.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:37.388+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:37.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:37.681+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:37.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:37.681+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:37.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:37.681+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:37.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:37.681+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:38.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:38.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:38.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:38.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:37.990+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:38.394+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:38.394+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:38.394+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:38.395+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:39.005 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:38.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:38.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:38.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:38.704+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:39.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:39.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:39.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:39.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:39.419+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:39.419+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:39.419+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:39.420+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:39.990 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:39.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.990 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:39.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.990 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:39.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:39.990 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:39.707+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:39.989+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:40.711 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:40.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:40.711 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:40.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:40.711 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:40.425+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:40.711 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:40.425+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:40.966 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:40.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:40.966 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:40.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:40.967 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:40.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:40.967 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:40.710+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:40.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:40.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:40.965+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:40.966+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:41.731 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:41.450+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:41.731 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:41.450+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:41.731 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:41.450+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:41.731 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:41.451+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:42.115 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:41.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:41.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:41.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:41.945+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:41.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:41.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:41.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.116 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:41.731+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:42.113+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8454 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:42.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:42.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:42.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:42.459+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:42.460+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:42.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:42.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:42.682+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:42.683+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:42.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:42.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:42.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:42.984+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:43.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:43.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:43.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:43.698+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:43.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:43.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:43.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:43.491+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:43.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:43.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:44.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:43.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:43.989+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:44.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:44.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:44.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:44.477+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:44.708+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:44.708+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:44.708+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:44.709+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:45.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:45.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:45.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:45.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:45.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:45.487+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:45.487+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:45.487+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:45.488+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:45.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:45.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:45.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:45.679+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:46.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:46.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:46.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:46.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:46.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:46.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:46.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:46.479+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:46.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:46.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:46.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:46.668+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:47.114+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8548 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:47.717 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:47.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:47.718 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:47.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:47.718 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:47.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:47.718 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:47.453+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:47.988 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:47.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:47.988 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:47.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:47.988 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:47.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:47.988 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:47.717+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.987+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.987+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.987+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:47.987+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:48.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:48.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:48.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:48.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:48.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:48.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:48.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:48.685+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:48.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:48.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:48.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:48.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:48.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:48.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:48.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:48.438+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:48.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:48.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:48.945+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:48.946+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:49.921 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:49.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:49.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:49.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:49.641+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:49.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:49.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:49.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:49.922 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:49.484+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:50.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:49.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:50.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:49.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:50.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:49.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:50.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:49.921+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:50.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:50.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:50.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:50.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:50.647+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:50.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:50.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:50.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:50.530+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:50.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:50.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:51.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:50.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:50.926+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:51.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:51.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:51.490+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:51.491+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:51.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:51.901 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:51.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:51.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:51.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:51.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:51.607+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:52.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:52.115+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8640 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:51.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:51.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:51.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:51.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:52.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:52.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:52.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:52.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:52.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:52.476+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:52.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:52.477+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:52.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:52.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:52.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:52.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:52.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:52.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:52.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:52.583+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:53.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:52.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:53.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:52.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:53.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:52.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:53.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:52.889+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:53.628+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:53.628+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:53.628+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:53.629+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:53.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:53.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:53.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:53.790 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:53.479+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:54.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:53.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:54.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:53.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:54.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:53.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:54.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:53.857+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:54.652+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:54.652+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:54.652+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:54.653+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:54.436+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:54.436+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:54.436+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:54.856 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:54.437+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:55.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:54.854+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:55.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:54.854+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:55.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:54.854+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:54.855+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:55.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:55.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:55.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:55.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:55.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:55.440+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:55.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:55.441+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:56.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:55.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:56.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:55.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:55.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:55.891+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:55.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:55.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:55.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:55.698+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:56.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:56.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:56.403+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:56.403+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:57.118 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:56.710+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:57.118 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:56.710+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:57.118 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:56.710+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:57.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:56.711+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:57.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:56.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:57.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:56.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:57.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:56.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:57.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:56.921+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:57.370 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:38:57.116+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8736 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:38:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:57.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:57.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:57.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:57.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:57.369+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:57.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:57.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:57.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:57.925+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:57.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:57.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:57.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:57.700+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:58.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:58.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:58.372+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:58.373+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:59.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:58.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:58.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:58.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:58.700+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:58.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:58.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:58.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:58.924+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:59.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:59.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:59.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:59.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:59.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:59.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:38:59.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:38:59.328+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:59.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:59.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:59.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:38:59.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:38:59.906+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:38:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:59.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:59.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:59.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:38:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:38:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:38:59.669+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:00.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:00.329+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:00.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:00.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:00.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:00.896+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:00.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:00.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:00.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:00.704+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:01.669 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:01.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:01.669 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:01.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:01.669 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:01.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:01.669 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:01.358+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:01.720+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:01.720+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:01.720+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:01.721+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:01.862+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:01.862+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:01.862+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:01.863+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:02.385 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:02.118+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8829 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:02.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:02.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:02.383+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:02.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:02.384+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:03.005 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:02.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:02.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:02.689+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:02.690+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:02.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:02.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:02.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:03.006 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:02.833+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:03.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:03.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:03.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:03.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:03.341+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:03.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:03.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:03.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:03.827+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:03.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:03.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:03.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:03.686+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:04.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:04.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:04.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:04.354+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:05.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:04.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:05.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:04.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:05.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:04.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:04.718+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:04.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:04.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:04.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:04.824+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:05.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:05.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:05.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:05.388+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:05.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:05.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:05.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:05.850+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:05.753+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:05.753+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:05.753+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:06.079 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:05.754+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:06.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:06.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:06.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:06.359+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:07.120 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:06.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:07.120 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:06.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:06.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:06.829+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:06.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:06.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:06.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:07.121 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:06.727+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:07.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:07.119+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 8925 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:07.351+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:07.351+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:07.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:07.351+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:07.352+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:07.810+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:07.810+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:07.810+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:07.811+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:07.734+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:07.734+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:07.734+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:07.735+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:08.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:08.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:08.374+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:08.375+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:09.150 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:08.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:09.150 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:08.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:09.150 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:08.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:09.150 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:08.813+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:09.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:08.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:09.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:08.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:09.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:08.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:09.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:08.757+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:09.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:09.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:09.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:09.400+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:09.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:10.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:09.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:09.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:09.833+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:09.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:09.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:09.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:09.771+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:10.407+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:10.407+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:10.407+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:10.408+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:11.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:10.787+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:10.787+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:10.787+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:10.788+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:10.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:10.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:10.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:10.851+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:11.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:11.360+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:11.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:11.360+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:11.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:11.360+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:11.670 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:11.361+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:11.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:11.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:11.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:11.793+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:11.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:11.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:11.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:11.845+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:12.373 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:12.120+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9014 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:12.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:12.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:12.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:12.372+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:13.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:12.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:13.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:12.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:12.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:12.794+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:12.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:12.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:12.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:12.872+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:13.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:13.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:13.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:13.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:13.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:13.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:13.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:13.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:13.907+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:13.908+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:14.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:13.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:14.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:13.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:14.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:13.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:14.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:13.782+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:14.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:14.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:14.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:14.320+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:14.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:14.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:14.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:14.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:14.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:14.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:14.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:14.762+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:15.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:15.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:15.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:15.349+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:15.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:15.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:15.954+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:15.955+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:15.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:15.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:15.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:15.781+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:16.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:16.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:16.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:16.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:16.313+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:16.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:16.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:16.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:16.944+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:16.759+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:16.759+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:16.759+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:17.123 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:16.760+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:17.122+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9108 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:17.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:17.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:17.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:17.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:17.302+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:17.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:17.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:17.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:17.922+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:17.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:17.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:17.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:17.792+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:18.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:18.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:18.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:18.290+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:19.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:18.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:18.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:18.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:18.833+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:18.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:18.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:18.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:18.890+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:19.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:19.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:19.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:19.259+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:20.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:19.940+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:19.860+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:19.860+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:19.860+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:20.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:19.861+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:20.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:20.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:20.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:20.241+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:20.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:20.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:20.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:20.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:20.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:20.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:20.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:20.907+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:21.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:21.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:21.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:21.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:21.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:21.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:21.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:21.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:21.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:21.858+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:21.858+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:21.858+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:21.858+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:22.247 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:22.122+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9193 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:22.248 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:21.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:22.248 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:21.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:22.248 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:21.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:22.248 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:21.949+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:22.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:22.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:22.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:22.323+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:23.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:22.953+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:22.953+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:22.953+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:22.954+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:22.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:22.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:22.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:22.904+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:23.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:23.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:23.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:23.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:23.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:23.361+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:24.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:23.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:24.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:23.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:24.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:23.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:24.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:23.904+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:23.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:23.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:23.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:23.943+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:24.333+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:24.333+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:24.333+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:24.334+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:24.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:24.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:24.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:24.967+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:24.892+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:24.892+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:24.892+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:25.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:24.893+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:25.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:25.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:25.329+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:25.330+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:25.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:25.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:25.963+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:25.963+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:25.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:25.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:25.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:26.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:25.937+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:26.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:26.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:26.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:26.301+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:27.249 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:27.123+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9280 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:27.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:27.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:27.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:27.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:27.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:27.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:27.250 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:27.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:27.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:26.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:27.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:26.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:27.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:26.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:27.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:26.985+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:27.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:27.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:27.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:27.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:27.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:27.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:27.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:27.322+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:28.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:28.027+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:28.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:28.027+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:28.027+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:28.027+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:28.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:28.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:28.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:28.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:28.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:28.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:28.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:28.274+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.015+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:29.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:29.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:29.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:29.076+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:29.234+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:29.234+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:29.234+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:29.235+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:29.985+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:30.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:30.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:30.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:30.090+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:30.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:30.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:30.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:30.227+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:31.259 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:30.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:30.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:30.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:30.941+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:31.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:31.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:31.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:31.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:31.109+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:31.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:31.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:31.672 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:31.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:31.672 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:31.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:31.672 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:31.259+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:32.250 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:32.124+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9368 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:31.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:31.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:31.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:31.989+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:32.218+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:32.218+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:32.218+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:32.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:32.218+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:32.252 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:32.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:32.252 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:32.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:32.252 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:32.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:32.252 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:32.151+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:33.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:33.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:33.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:33.153+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:32.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:32.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:32.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:33.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:32.979+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:33.561 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:33.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:33.562 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:33.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:33.562 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:33.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:33.562 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:33.238+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:34.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:33.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:34.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:33.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:33.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:33.984+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:34.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:34.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:34.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:34.191+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:34.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:34.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:34.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:34.226+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:34.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:34.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:34.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:34.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:35.207+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:35.207+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:35.207+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:35.208+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:35.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:35.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:35.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:35.232+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:36.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:36.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:36.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:36.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:36.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:36.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:36.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:36.227+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:36.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:36.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:36.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:36.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:36.263+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:37.125+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9460 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:37.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:37.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:37.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:37.226+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:37.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:37.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:37.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:37.276+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:37.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:37.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:37.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:37.042+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:38.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:38.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:38.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:38.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:38.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:38.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:38.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:38.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:38.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:38.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:38.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:38.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:38.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:38.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:38.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:38.200+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:38.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:38.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:38.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:38.318+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:39.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:39.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:39.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:39.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:39.222+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:39.222+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:39.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:39.222+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:39.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:39.223+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:39.301+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:40.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:40.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:40.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:40.265+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:40.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:40.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:40.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:40.064+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:40.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:40.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:40.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:40.204+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:41.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:41.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:41.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:41.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:41.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:41.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:41.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:41.206+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:41.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:41.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:41.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:41.241+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:42.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:42.126+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9551 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:42.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:42.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:42.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:42.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:42.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:42.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:42.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:42.010+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:42.293 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:42.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:42.293 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:42.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:42.293 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:42.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:42.293 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:42.235+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:42.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:42.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:42.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:42.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:43.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:43.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:43.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:43.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:43.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:43.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:43.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:43.056+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:43.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:43.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:43.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:43.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:43.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:43.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:43.563 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:43.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:43.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:43.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:43.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:43.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:43.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:43.258+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:43.563 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:43.258+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:44.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:44.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:44.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:44.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:44.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:44.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:44.305 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:44.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:44.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:44.224+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:44.303+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:44.303+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:44.303+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:44.304+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:45.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:45.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:45.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:45.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:45.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:45.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:45.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:45.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:45.202+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:45.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:45.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:45.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:45.079+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:46.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:46.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:46.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:46.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:46.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:46.070+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:46.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:46.071+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:46.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:46.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:46.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:46.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:46.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:46.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:46.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:46.155+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:46.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:46.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:46.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:46.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:47.127+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9647 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:47.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:47.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:47.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:47.108+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:47.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:47.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:47.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:47.146+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:47.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:47.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:47.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:47.283+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:48.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:48.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:48.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:48.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:48.146+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:48.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:48.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:48.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:48.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:48.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:48.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:48.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:48.254+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:49.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:49.101+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:49.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:49.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:49.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:49.143+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:49.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:49.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:49.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:49.239+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:50.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:50.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:50.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:50.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:50.056+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:50.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:50.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:50.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:50.183+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:50.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:50.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:50.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:50.231+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:51.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:51.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:51.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:51.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:51.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:51.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:51.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:51.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:51.257+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:51.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:51.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:51.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:51.066+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:52.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:52.128+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9739 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:52.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:52.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:52.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:52.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:52.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:52.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:52.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:52.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:52.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:52.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:52.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:52.225+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:53.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:53.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:53.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:53.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:53.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:53.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:53.268 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:53.003+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:53.563 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:53.563 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:53.563 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:53.563 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:53.267+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:53.564 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:53.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:53.564 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:53.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:53.564 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:53.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:53.564 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:53.274+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:54.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:54.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:54.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:54.237+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:54.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:54.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:54.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:54.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:55.229 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:55.229 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:55.230 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.964+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:55.230 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:54.964+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:55.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:55.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:55.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:55.250+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:55.292+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:56.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:55.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:56.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:55.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:55.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:55.923+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:56.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:56.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:56.274+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:56.275+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:56.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:56.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:56.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:56.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:57.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:39:57.130+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9834 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:39:57.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:56.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:57.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:56.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:56.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:56.905+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:57.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:57.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:57.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:57.317+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:57.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:57.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:57.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:57.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:57.288+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:57.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:57.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:57.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:57.877+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:58.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:58.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:58.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:58.282+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:58.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:58.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:58.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:58.293+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:59.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:58.853+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:59.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:58.853+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:58.853+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:39:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:58.853+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:59.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:59.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:59.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:59.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:39:59.277+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:59.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:59.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:59.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:39:59.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:39:59.300+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:00.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:59.808+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:00.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:59.808+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:00.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:59.808+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:00.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:39:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:39:59.808+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:00.239+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:00.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:00.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:00.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:00.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:00.316+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:00.761+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:00.761+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:00.761+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:00.762+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:01.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:01.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:01.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:01.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:01.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:01.673 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:01.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:01.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:01.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:01.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:01.318+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:01.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:01.804+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:01.804+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:01.804+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:01.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:01.805+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:02.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:02.132+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 9927 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:02.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:02.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:02.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:02.215+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:02.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:02.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:02.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:02.335+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:03.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:02.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:02.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:02.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:02.850+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:03.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:03.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:03.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:03.343+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:03.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:03.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:03.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:03.171+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:04.142 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:03.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:04.142 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:03.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:04.142 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:03.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:04.142 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:03.873+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:04.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:04.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:04.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:04.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:04.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:04.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:04.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:04.141+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:04.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:04.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:04.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:04.391+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:05.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:04.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:05.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:04.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:05.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:04.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:05.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:04.876+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:05.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:05.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:05.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:05.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:05.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:05.170+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:05.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:05.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:05.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:05.391+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:05.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:05.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:05.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:05.911+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:06.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:06.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:06.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:06.155+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:06.384+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:06.384+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:06.384+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:06.385+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:07.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:07.132+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10022 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:06.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:06.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:06.924+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:06.924+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:07.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:07.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:07.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:07.118+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:07.518 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:07.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:07.518 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:07.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:07.518 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:07.358+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:07.518 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:07.358+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:08.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:08.099+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:08.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:08.099+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:08.099+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:08.099+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:07.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:07.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:07.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:07.914+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:08.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:08.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:08.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:08.398+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:09.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:08.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:09.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:08.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:09.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:08.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:09.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:08.942+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:09.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:09.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:09.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:09.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:09.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:09.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:09.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:09.093+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:09.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:09.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:09.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:09.398+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:10.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:10.074+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:09.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:09.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:09.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:09.901+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:10.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:10.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:10.385+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:10.385+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:11.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:11.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:11.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:11.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:11.092+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:11.093+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:10.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:10.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:10.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:10.855+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:11.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:11.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:11.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:11.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:11.674 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:11.371+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:11.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:11.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:11.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:11.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:11.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:12.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:12.133+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10116 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:12.363 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:12.057+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:12.363 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:12.057+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:12.363 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:12.057+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:12.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:12.058+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:12.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:12.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:12.361+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:12.362+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:12.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:12.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:12.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:12.865+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:13.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:13.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:13.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:13.022+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:13.661 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:13.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:13.662 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:13.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:13.662 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:13.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:13.662 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:13.368+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:13.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:13.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:13.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:13.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:13.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:13.884+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:14.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:14.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:14.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:14.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:14.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:14.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:14.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:14.006+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:14.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:14.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:14.365+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:14.366+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:14.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:14.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:14.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:14.837+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:15.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:15.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:15.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:15.024+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:15.359+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:15.359+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:15.359+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:15.360+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:15.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:15.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:15.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:15.816+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:16.052+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:16.052+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:16.052+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:16.053+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:16.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:16.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:16.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:16.392+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:17.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:16.802+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:17.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:16.802+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:17.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:16.802+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:17.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:16.803+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:17.137 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:17.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:17.137 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:17.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:17.137 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:17.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:17.137 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:17.005+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:17.134+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10214 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:17.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:17.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:17.367+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:17.368+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:18.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:18.055+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:18.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:18.055+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:18.055+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:18.055+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:17.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:17.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:17.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:17.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:18.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:18.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:18.340+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:18.341+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:19.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:18.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:19.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:18.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:19.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:18.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:19.107 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:18.856+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:19.368 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:19.105+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:19.368 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:19.105+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:19.368 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:19.105+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:19.369 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:19.106+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:19.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:19.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:19.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:19.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:19.367+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:20.114 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:19.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:20.114 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:19.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:20.114 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:19.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:20.114 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:19.819+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:20.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:20.112+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:20.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:20.112+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:20.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:20.112+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:20.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:20.113+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:20.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:20.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:20.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:20.362+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:20.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:20.363+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:21.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:20.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:20.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:20.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.161 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:20.835+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:21.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:21.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:21.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:21.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:21.160+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:21.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:21.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:21.376+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:21.377+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:21.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:21.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:21.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:21.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:21.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:21.838+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:22.406 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:22.135+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10308 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:22.406 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:22.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:22.406 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:22.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:22.406 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:22.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:22.406 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:22.208+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:22.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:22.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:22.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:22.404+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:22.405+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:23.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:22.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:23.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:22.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:22.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:22.874+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:23.565 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:23.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:23.565 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:23.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:23.565 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:23.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:23.565 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:23.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:23.565 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:23.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:23.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:23.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:23.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:23.448+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:23.566 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:23.449+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:23.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:23.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:23.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:23.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:24.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:24.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:24.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:24.298+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:24.408+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:24.408+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:24.408+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:24.409+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:25.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:24.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:25.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:24.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:24.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:24.866+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:25.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:25.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:25.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:25.279+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:25.386+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:25.386+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:25.386+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:25.386+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:26.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:25.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:26.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:25.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:25.823+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:25.824+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:26.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:26.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:26.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:26.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:26.269+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:26.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:26.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:26.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:26.434+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:27.138 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:26.815+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:27.138 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:26.815+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:27.138 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:26.815+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:27.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:26.816+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:27.397 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:27.137+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10403 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:27.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:27.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:27.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:27.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:27.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:27.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:27.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:27.300+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:27.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:27.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:27.396+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:27.396+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:27.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:27.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:27.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:27.793+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:28.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:28.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:28.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:28.336+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:28.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:28.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:28.395+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:28.395+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:29.022 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:28.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:29.023 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:28.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:29.023 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:28.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:29.023 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:28.793+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:29.350+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:29.350+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:29.350+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:29.351+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:29.328+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:29.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:29.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:29.329+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:30.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:29.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:30.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:29.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:30.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:29.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:30.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:29.833+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:30.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:30.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:30.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:30.329+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:30.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:30.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:30.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:30.336+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:30.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:30.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:30.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:30.878+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:31.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:31.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:31.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:31.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:31.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:31.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:31.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:31.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:31.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:31.338+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:32.093 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:31.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:32.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:31.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:32.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:31.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:32.094 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:31.923+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:32.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:32.138+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10494 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:32.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:32.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:32.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:32.337+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:32.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:32.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:32.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:32.302+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:32.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:32.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:32.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:32.973+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:33.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:33.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:33.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:33.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:33.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:33.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:33.566 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:33.328+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:33.567 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:33.567 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:33.567 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:33.567 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:33.297+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:34.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:33.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:34.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:33.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:34.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:33.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:34.310 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:33.996+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:34.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:34.362+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:34.362+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:34.362+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:34.362+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:34.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:34.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:34.308+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:34.309+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:35.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:35.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:35.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.017+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:35.324 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.018+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:35.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:35.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:35.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:35.323+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:35.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:35.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:35.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:35.391+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:36.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:36.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:36.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:36.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:35.990+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:36.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:36.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:36.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:36.288+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:36.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:36.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:36.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:36.413+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:37.139+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10589 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:37.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:37.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:37.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:37.019+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:37.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:37.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:37.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:37.251+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:37.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:37.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:37.424+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:37.425+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:38.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:38.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:38.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:38.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:38.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:38.251+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:38.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:38.060+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:38.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:38.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:38.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:38.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:38.398+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:39.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:39.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:39.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:39.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:39.045+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:39.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:39.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:39.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:39.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:39.423+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:39.300+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:39.301+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:40.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:40.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:40.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:40.066+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:40.266+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:40.266+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:40.266+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:40.267+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:40.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:40.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:40.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:40.468+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:40.468+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:41.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:41.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:41.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.031+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:41.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.032+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:41.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:41.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:41.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:41.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:41.421+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:41.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:41.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:41.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:41.297+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:42.264 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:42.140+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10682 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:42.264 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:42.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:42.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:42.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:41.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:42.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:42.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:42.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:42.301+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:42.302+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:42.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:42.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:42.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:42.451+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:43.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:43.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:43.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:43.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:43.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:43.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:43.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:43.004+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:43.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:43.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:43.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:43.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:43.435+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:43.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:43.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:43.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:43.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:43.318+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:44.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:44.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:44.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:44.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:44.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:44.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:44.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:44.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:44.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:44.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:44.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:44.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:44.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:44.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:44.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:44.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:44.329+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:45.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:45.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:45.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:45.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.014+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:45.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:45.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:45.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:45.371+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:45.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:45.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:45.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:45.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:45.375+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:45.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:45.376+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:46.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:46.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:46.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.979+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:46.249 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:45.980+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:46.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:46.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:46.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:46.378+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:46.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:46.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:46.371+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:46.372+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:47.266 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:47.143+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10776 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:47.266 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:46.971+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:47.266 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:46.971+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:47.266 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:46.971+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:47.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:46.971+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:47.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:47.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:47.346+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:47.347+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:47.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:47.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:47.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:47.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:47.354+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:48.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:48.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:48.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.013+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:48.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.014+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:48.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:48.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:48.366+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:48.367+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:48.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:48.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:48.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:48.323+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:49.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:49.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:49.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:49.292 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:48.978+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:49.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:49.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:49.381+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:49.382+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:49.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:49.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:49.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:49.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:49.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:50.286 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:50.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:50.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:50.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:50.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:50.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:50.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:50.027+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:50.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:50.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:50.370+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:51.365 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:51.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:51.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:51.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:51.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:51.237+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:51.237+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:51.237+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:51.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:51.238+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:51.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:51.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:51.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:51.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:51.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:51.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:52.347 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:52.143+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10869 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:52.347 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:52.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:52.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:52.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:52.284+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:52.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:52.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:52.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:52.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:52.054+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:52.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:52.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:52.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:52.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:52.346+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:52.347+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:53.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:53.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:53.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:53.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:53.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:53.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:53.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:53.029+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:53.567 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:53.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:53.567 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:53.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:53.567 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:53.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:53.567 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:53.306+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:53.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:53.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:53.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:53.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:53.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:53.377+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:53.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:53.378+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:54.349 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:54.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:54.349 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:54.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:54.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:54.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:54.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:54.075+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:54.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:54.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:54.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:54.348+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:54.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:54.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:54.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:54.371+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:55.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:55.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:55.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:55.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:55.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:55.080+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:55.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:55.080+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:55.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:55.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:55.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:55.357+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:55.363+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:55.363+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:55.363+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:55.364+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:56.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:56.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:56.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:56.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:56.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:56.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:56.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:56.048+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:56.381+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:56.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:56.381+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:56.381+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:56.382+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:56.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:56.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:56.368+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:56.369+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:57.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:40:57.144+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 10965 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:40:57.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:57.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:57.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:57.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:57.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:57.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:57.052+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:57.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:57.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:57.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:57.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:57.400+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:57.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:57.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:57.411+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:57.412+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:58.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:58.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:58.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:58.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:58.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:58.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:58.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:58.421+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:58.422+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:58.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:58.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:58.453+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:58.454+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:40:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:58.982+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:59.466+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:59.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:59.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:59.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:40:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:40:59.467+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:40:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:59.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:59.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:59.463+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:40:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:40:59.464+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:59.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:59.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:59.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:40:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:40:59.957+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:00.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:00.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:00.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:00.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:00.432+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:00.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:00.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:00.467+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:00.468+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:01.415 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:00.967+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:01.415 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:00.967+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:01.415 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:00.967+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:01.415 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:00.968+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:01.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:01.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:01.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:01.414+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:01.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:01.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:01.433+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:01.434+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:02.265 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:02.145+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11059 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:02.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:01.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:02.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:01.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:02.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:01.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:02.265 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:01.932+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:02.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:02.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:02.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:02.444+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:02.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:02.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:02.469+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:02.470+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:02.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:02.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:02.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:03.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:02.913+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:03.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:03.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:03.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:03.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:03.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:03.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:03.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:03.454+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:03.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:03.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:03.460+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:03.461+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:03.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:03.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:03.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:03.923+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:04.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:04.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:04.417+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:04.418+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:04.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:04.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:04.504+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:04.505+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:04.952+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:04.952+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:04.952+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:04.952+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:05.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:05.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:05.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:05.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:05.546+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:05.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:06.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:05.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:05.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:05.948+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:06.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:06.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:06.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:06.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:06.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:06.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:06.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:06.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:06.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:06.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:06.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:06.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:06.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:06.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:06.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:06.572+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:07.159 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:07.147+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11153 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:07.159 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:06.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:06.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:06.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:07.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:06.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:07.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:07.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:07.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:07.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:07.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:07.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:07.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:07.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:07.551+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:07.857 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:07.552+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:07.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:07.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:07.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:07.856+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:08.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:08.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:08.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:08.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:08.417+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:08.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:08.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:08.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:08.535+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:08.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:08.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:08.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:08.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:08.959 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:08.867+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:09.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:09.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:09.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:09.536+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:09.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:09.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:09.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:09.420+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:10.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:09.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:10.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:09.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:10.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:09.850+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:10.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:09.851+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:10.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:10.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:10.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:10.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:10.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:10.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:10.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:10.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:10.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:10.835+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:10.835+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:10.835+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:11.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:10.835+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:11.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:11.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:11.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:11.541+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:11.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:11.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:11.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:11.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:11.384+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:12.029 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:11.800+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:12.030 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:11.800+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:12.030 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:11.800+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:12.030 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:11.800+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:12.349 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:12.148+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11247 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:12.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:12.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:12.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:12.533+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:12.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:12.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:12.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:12.348+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:12.349+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:12.758+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:12.758+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:12.758+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:13.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:12.759+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:13.530+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:13.530+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:13.530+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:13.531+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:13.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:13.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:13.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:13.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:13.345+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:13.805+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:14.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:13.805+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:13.805+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:14.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:13.806+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:14.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:14.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:14.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:14.509+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:14.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:14.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:14.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:14.380+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:14.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:14.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:14.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:14.852+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:15.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:15.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:15.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:15.416+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:15.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:15.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:15.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:15.508+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:15.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:15.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:15.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:15.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:16.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:16.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:16.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:16.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:16.423+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:16.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:16.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:16.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:16.554+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:17.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:16.790+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:17.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:16.790+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:17.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:16.790+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:17.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:16.791+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:17.149+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11343 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:17.782 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:17.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:17.782 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:17.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:17.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:17.445+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:17.528+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:17.528+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:17.528+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:17.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:17.529+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:17.781+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:17.781+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:17.781+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:17.782+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:18.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:18.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:18.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:18.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:18.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:18.818 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:18.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:18.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:18.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:18.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:18.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:18.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:18.563+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:19.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:18.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:19.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:18.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:18.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:18.817+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:19.788 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:19.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:19.788 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:19.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:19.788 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:19.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:19.788 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:19.490+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:19.789 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:19.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:19.789 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:19.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:19.789 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:19.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:19.789 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:19.587+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:20.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:19.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:20.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:19.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:20.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:19.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:20.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:19.788+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:20.803 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:20.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:20.803 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:20.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:20.803 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:20.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:20.803 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:20.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:20.804 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:20.580+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:20.804 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:20.580+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:20.804 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:20.580+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:20.804 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:20.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:20.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:20.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:21.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:20.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:20.802+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:21.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:21.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:21.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:21.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:21.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:21.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:21.757 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:21.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:21.758 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:21.603+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:22.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:21.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:22.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:21.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:22.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:21.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:22.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:21.756+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:22.150+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11437 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:22.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:22.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:22.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:22.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:22.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:22.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:22.805 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:22.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:22.806 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:22.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:22.806 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:22.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:22.806 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:22.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:22.806 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:22.641+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:23.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:22.803+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:23.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:22.803+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:22.803+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:22.804+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:23.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:23.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:23.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:23.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:23.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:23.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:23.818 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:23.543+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:23.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:23.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:23.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:23.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:23.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:23.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:23.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:23.622+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:23.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:23.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:23.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:23.817+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:24.579 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:24.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:24.579 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:24.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:24.579 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:24.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:24.579 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:24.556+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:24.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:24.811+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:24.811+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:24.811+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:24.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:24.812+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:24.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:24.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:24.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:24.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:24.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:24.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:24.578+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:25.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:25.821+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:25.821+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:25.821+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:25.822+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:25.576+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:25.576+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:25.576+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:25.576+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:25.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:25.602+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:26.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:26.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:26.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:26.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:26.554+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:26.852 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:26.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:26.852 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:26.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:26.852 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:26.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:26.852 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:26.558+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:27.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:26.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:27.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:26.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:27.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:26.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:27.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:26.850+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:27.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:27.152+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11531 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:27.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:27.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:27.596+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:27.597+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:27.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:27.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:27.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:27.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:27.851 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:27.568+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:27.848+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:27.848+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:27.848+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:27.849+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:28.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:28.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:28.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:28.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:28.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:28.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:28.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:28.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:28.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:28.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:28.581+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:29.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:28.891+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:29.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:28.891+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:29.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:28.891+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:28.892+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:29.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:29.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:29.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:29.625+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:29.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:29.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:29.881 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:29.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:29.882 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:29.667+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:30.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:29.879+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:30.147 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:29.879+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:30.148 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:29.879+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:30.148 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:29.880+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:30.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:30.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:30.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:30.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:30.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:30.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:30.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:30.656+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:30.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:30.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:30.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:30.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:30.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:30.655+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:30.865 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:30.656+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:30.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:30.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:30.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:30.865+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:31.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:31.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:31.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:31.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:31.695+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:31.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:31.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:31.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:31.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:31.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:31.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:31.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:31.655+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:32.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:32.154+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11624 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:33.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:32.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:32.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:32.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:32.698+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:32.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:32.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:32.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:32.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:32.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:32.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:32.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:32.716+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:33.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:33.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:33.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:33.655+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:33.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:33.656+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:33.840 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:33.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.841 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:33.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.841 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:33.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:33.841 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:33.760+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:34.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:33.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:33.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:33.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:33.839+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:34.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:34.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:34.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:34.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:34.664+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:34.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:34.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:34.816+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:34.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:34.816+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:34.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:34.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:34.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:34.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:34.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:34.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:34.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:34.735+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:35.692+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:35.692+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:35.692+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:35.692+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:35.806+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:35.806+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:35.806+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:35.807+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:35.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:35.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:35.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:35.725+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:36.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:36.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:36.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:36.655+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:36.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:36.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:36.834+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:36.835+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:36.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:36.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:36.705+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:36.705+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:37.154+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11720 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:37.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:37.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:37.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:37.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:37.649+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:37.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:37.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:37.801+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:37.802+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:37.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:37.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:37.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:37.733+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:38.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:38.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:38.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:38.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:38.820+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:38.820+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:38.820+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:38.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:38.821+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:38.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:38.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:38.609+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:38.610+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:39.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:39.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:39.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:39.864+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:39.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:39.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:39.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:39.659+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:39.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:39.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:39.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:39.791+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:40.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:40.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:40.636+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:40.637+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:40.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:40.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:40.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:40.760+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:40.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:40.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:40.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:40.883+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:41.638+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:41.638+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:41.638+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:41.639+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:41.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:41.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:41.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:41.780+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:41.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:41.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:41.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:41.865+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:42.156+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11814 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:43.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:42.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:42.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:42.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:42.799+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:42.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:42.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:42.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:42.681+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:42.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:42.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:42.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:42.879+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:43.884 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:43.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.884 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:43.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.884 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:43.706+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:43.884 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:43.707+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:43.885 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:43.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.885 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:43.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.885 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:43.805+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:43.885 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:43.806+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:43.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:43.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:43.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:43.883+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:45.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:44.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:44.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:44.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:44.745+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:44.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:44.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:44.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:44.898+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:45.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:44.855+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:44.855+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:44.855+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:44.856+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:45.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:46.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:45.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:45.942+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:45.943+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:45.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:45.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:45.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:45.869+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:45.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:45.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:46.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:45.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:46.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:45.758+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:47.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:46.752+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:47.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:46.752+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:47.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:46.752+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:47.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:46.753+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:46.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:46.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:46.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:46.911+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:46.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:46.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:46.966+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:47.160 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:46.967+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:47.157+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 11907 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:48.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:47.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:47.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:47.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:47.712+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:47.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:47.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:47.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:47.915+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:48.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:47.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:48.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:47.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:48.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:47.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:48.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:47.949+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:48.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:48.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:48.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:48.667+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:48.668+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:48.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:48.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:48.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:48.924+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:48.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:48.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:48.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:48.929+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:49.947 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:49.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:49.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:49.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:49.946+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:49.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:49.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:49.914+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:49.915+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:49.948 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:49.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:49.949 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:49.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:49.949 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:49.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:49.949 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:49.696+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:51.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:50.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:50.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:50.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:50.911+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:50.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:50.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:50.744+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:50.745+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:51.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:50.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:51.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:50.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:51.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:50.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:51.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:50.975+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:51.942 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:51.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.942 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:51.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.942 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:51.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:51.942 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:51.898+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:51.943 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:51.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.943 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:51.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.943 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:51.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:51.943 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:51.784+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:52.282 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:52.158+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12001 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:52.282 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:51.940+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:52.283 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:51.940+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:52.283 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:51.940+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:52.283 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:51.941+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:53.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:52.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:52.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:52.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:52.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:52.746+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:52.746+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:52.746+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:52.747+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:52.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:52.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:52.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:52.967+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:53.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:53.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:53.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:53.717+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:53.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:53.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:53.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:53.928+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:53.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:53.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:53.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:53.976+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:54.923 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:54.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:54.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:54.921+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:54.922+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:54.671+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:54.671+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:54.671+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:54.924 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:54.672+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:55.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:55.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:55.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:55.027+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:56.068 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:55.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:56.068 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:55.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:56.068 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:55.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:56.068 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:55.916+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:56.069 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:55.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:56.069 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:55.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:56.069 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:55.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:56.069 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:55.713+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:56.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:56.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:56.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:56.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:56.067+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:57.062 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:56.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:57.062 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:56.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:57.062 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:56.906+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:57.062 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:56.907+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:57.063 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:56.727+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:57.063 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:56.727+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:57.063 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:56.727+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:57.063 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:56.728+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:57.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:41:57.160+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12098 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:41:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:57.060+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:57.060+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:57.060+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:57.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:57.061+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:58.056 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:57.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:58.056 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:57.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:58.056 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:57.684+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:58.056 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:57.685+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:58.057 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:57.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:58.057 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:57.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:58.057 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:57.897+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:58.057 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:57.898+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:58.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:58.054+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:58.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:58.054+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:58.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:58.054+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:58.055+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:58.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:58.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:58.910+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:58.910+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:58.715+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:59.011 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:58.715+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:59.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:58.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:41:59.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:58.717+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:41:59.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:59.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:59.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:59.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:41:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:41:59.010+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:00.011 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:59.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:59.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:59.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:41:59.941+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:59.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:59.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:59.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:00.012 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:41:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:41:59.751+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:00.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:00.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:00.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:00.011+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:00.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:00.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:00.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:00.937+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:00.786+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:00.786+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:00.786+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:00.787+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:01.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:01.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:01.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:01.045+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:01.959 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:01.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.959 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:01.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.959 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:01.784+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:01.959 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:01.785+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:02.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:01.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:01.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:01.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:01.958+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:02.050+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:02.050+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:02.050+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:02.051+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:02.288 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:02.161+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12193 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:03.085 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:02.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:03.085 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:02.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:03.085 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:02.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:03.086 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:02.927+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:03.086 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:02.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.086 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:02.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.086 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:02.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.086 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:02.772+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:03.082+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:03.082+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:03.082+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:03.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:03.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:03.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:03.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:03.793+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:03.907 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:03.794+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:04.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:03.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:03.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:03.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:03.905+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:04.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:04.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:04.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:04.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:04.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:05.096 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:04.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:05.096 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:04.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:05.096 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:04.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:05.096 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:04.894+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:05.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:04.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:05.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:04.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:05.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:04.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:05.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:04.771+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:05.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:05.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:05.094+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:05.095+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:06.077 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:05.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:05.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:05.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:05.813+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:05.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:05.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:05.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:06.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:05.942+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:06.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:06.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:06.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:06.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:06.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:06.077+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:07.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:06.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:07.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:06.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:07.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:06.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:07.103 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:06.948+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:07.103 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:06.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:07.103 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:06.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:07.103 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:06.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:07.103 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:06.772+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:07.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:07.162+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12288 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:07.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:07.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:07.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:07.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:07.102+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:07.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:07.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:07.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:07.929+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:07.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:07.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:07.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:08.143 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:07.767+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:08.141+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:08.141+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:08.141+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:08.142+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:09.103 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:08.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:09.131 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:08.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:09.131 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:08.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:09.131 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:08.888+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:09.132 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:08.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:09.132 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:08.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:09.132 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:08.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:09.132 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:08.800+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:09.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:09.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:09.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:09.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:09.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:09.101+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:09.102+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:10.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:09.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:10.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:09.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:10.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:09.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:10.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:09.857+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:10.106 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:09.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:10.106 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:09.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:10.106 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:09.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:10.106 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:09.800+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:10.397 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:10.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:10.398 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:10.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:10.398 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:10.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:10.398 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:10.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:10.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:10.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:10.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:10.885+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:10.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:10.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:10.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:11.136 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:10.781+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:11.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:11.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:11.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:11.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:11.135+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:11.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:11.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:11.855+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:11.856+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:11.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:11.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:12.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:11.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:12.130 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:11.764+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:12.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:12.164+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12382 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:12.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:12.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:12.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:12.128+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:13.126 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:12.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:13.126 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:12.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:13.126 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:12.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:13.126 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:12.842+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:13.127 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:12.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:13.127 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:12.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:13.127 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:12.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:13.127 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:12.791+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:13.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:13.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:13.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:13.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:13.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:13.124+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:13.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:13.125+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:14.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:13.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:14.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:13.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:13.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:13.794+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:13.815+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:13.815+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:13.815+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:14.172 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:13.816+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:14.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:14.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:14.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:14.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:14.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:14.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:14.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:14.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:14.749+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:14.853+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:14.853+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:14.853+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:14.854+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:15.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:15.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:15.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:15.193+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:16.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:15.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:15.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:15.755+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:15.757+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:15.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:15.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:15.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:15.809+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:16.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:16.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:16.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:16.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:16.155+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:16.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:16.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:16.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:16.766+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:16.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:16.767+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:17.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:16.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:17.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:16.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:17.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:16.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:17.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:16.792+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:17.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:17.166+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12478 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:17.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:18.133 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:17.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:17.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:17.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:17.800+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:17.769+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:17.769+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:17.769+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:18.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:17.770+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:18.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:18.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:18.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:18.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:18.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:18.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:18.134+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:18.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:18.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:18.791+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:18.792+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:18.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:18.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:18.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:19.105 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:18.811+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:19.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:19.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:19.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:19.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:19.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:19.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:19.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:19.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:19.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:19.780+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:19.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:20.073 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:19.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:20.074 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:19.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:20.074 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:19.798+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:20.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:20.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:20.071+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:20.072+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:21.091 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:20.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:21.091 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:20.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:21.091 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:20.812+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:21.091 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:20.813+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:21.092 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:20.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:21.092 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:20.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:21.092 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:20.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:21.092 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:20.802+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:21.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:21.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:21.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:21.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:21.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:21.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:21.090+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:21.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:21.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:21.841+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:21.841+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:21.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:21.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:21.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:22.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:21.827+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:22.166+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12572 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:22.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:22.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:22.137+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:22.138+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:22.796+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:22.796+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:22.796+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:22.797+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:22.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:22.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:22.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:23.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:22.838+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:23.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:23.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:23.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:23.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:23.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:23.166+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:23.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:23.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:23.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:23.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:23.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:23.813+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:24.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:23.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:24.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:23.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:24.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:23.829+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:24.139 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:23.830+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:24.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-19T06:42:24.136+0000 7fa873b64700 0 monclient(hunting): authenticate timed out after 300 2024-01-19T06:42:24.140 INFO:teuthology.orchestra.run.smithi134.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-01-19T06:42:24.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:24.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:24.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:24.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:24.186+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T06:42:24.899 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-01-19T06:42:25.127 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cc05a3c497cb461fa448f060d4858251 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1402, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-01-19T06:42:25.131 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-01-19T06:42:25.177 INFO:tasks.kclient:Unmounting kernel clients... 2024-01-19T06:42:25.177 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:42:25.177 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:42:25.190 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:24.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:25.191 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:24.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:25.191 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:24.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:25.191 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:24.865+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:25.191 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:24.782+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:25.192 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:24.782+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:25.192 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:24.782+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:25.192 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:24.783+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:25.210 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:42:25.210 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:42:25.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T06:42:25.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:25.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:25.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:25.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:25.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:25.169+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:26.157 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:25.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:26.158 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:25.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:26.158 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:25.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:26.158 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:25.802+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:26.158 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:25.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:26.159 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:25.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:26.159 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:25.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:26.159 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:25.878+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:26.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:26.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:26.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:26.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:26.156+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:27.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:26.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:27.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:26.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:27.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:26.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:27.171 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:26.792+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:27.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:26.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:27.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:26.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:27.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:26.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:27.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:26.928+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:27.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:27.168+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12667 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:27.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:27.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:27.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:27.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:27.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:27.173+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:28.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:27.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:27.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:27.959+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:27.960+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:27.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:27.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:27.791+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:27.792+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:28.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:28.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:28.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:28.201+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:28.990 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:28.835+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:28.835+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:28.835+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:28.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:28.836+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:28.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:28.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:28.989+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:28.990+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:29.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:29.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:29.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:29.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:29.236+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:30.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:29.970+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:29.970+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:29.970+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:29.970+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:29.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:29.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:30.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:29.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:30.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:29.830+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:30.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:30.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:30.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:30.274+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:31.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:30.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:30.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:30.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:30.841+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:30.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:31.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:30.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:31.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:30.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:31.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:30.983+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:31.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:31.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:31.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:31.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:31.292+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:31.292+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:31.973 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:31.875+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.974 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:31.875+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.974 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:31.875+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:31.974 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:31.875+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:32.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:32.169+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12762 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:32.295 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:31.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:32.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:31.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:32.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:31.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:32.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:31.972+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:32.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:32.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:32.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:32.329+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:33.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:32.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:32.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:32.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:32.933+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:33.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:32.828+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:33.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:32.828+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:33.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:32.828+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:33.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:32.828+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:33.575 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:33.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:33.575 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:33.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:33.575 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:33.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:33.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:33.314+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:34.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:33.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:33.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:33.947+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:33.948+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:34.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:33.876+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:34.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:33.876+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:34.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:33.876+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:34.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:33.877+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:34.298+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:34.299+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:34.299+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:34.299+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:34.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:34.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:34.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:34.952+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:35.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:34.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:34.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:34.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:34.910+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:35.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:35.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:35.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:35.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:35.329+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:36.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:35.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:35.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:35.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:35.934+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:35.917+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:35.918+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:36.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:35.918+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:36.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:35.919+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:36.283+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:36.283+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:36.283+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:36.284+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:37.172 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:36.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:36.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:36.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:36.921+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:36.881+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:36.881+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:37.173 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:36.881+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:37.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:36.881+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:37.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:37.171+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12859 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:37.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:37.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:37.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:37.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:37.331+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:37.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:37.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:37.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:37.863+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:37.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:37.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:37.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:37.879+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:38.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:38.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:38.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:38.324+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:39.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:38.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:39.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:38.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:39.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:38.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:39.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:38.884+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:38.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:38.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:38.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:38.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:39.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:39.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:39.334+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:39.334+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:40.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:39.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:40.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:39.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:40.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:39.865+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:40.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:39.866+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:40.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:39.898+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:40.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:39.898+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:40.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:39.898+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:40.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:39.899+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:40.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:40.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:40.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:40.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:41.118 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:40.842+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:41.118 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:40.842+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:41.118 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:40.842+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:41.118 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:40.843+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:41.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:40.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:41.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:40.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:41.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:40.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:41.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:40.921+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:41.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:41.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:41.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:41.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:41.295+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:42.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:41.831+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:42.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:41.831+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:42.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:41.831+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:42.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:41.832+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:42.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:41.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:42.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:41.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:42.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:41.883+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:42.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:41.884+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:42.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:42.173+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 12952 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:42.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:42.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:42.273+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:42.274+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:43.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:42.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:42.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:42.877+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:42.878+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:42.861+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:42.861+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:42.861+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:42.861+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:43.575 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:43.299+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:43.575 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:43.299+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:43.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:43.299+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:43.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:43.300+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:43.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:43.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:43.878+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:43.878+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:44.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:43.844+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:43.844+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:43.844+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:43.844+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:44.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:44.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:44.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:44.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:45.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:44.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:44.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:44.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:44.885+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:44.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:44.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:45.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:44.838+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:45.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:44.839+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:45.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:45.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:45.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:45.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:45.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:46.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:45.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:45.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:45.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:45.921+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:45.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:45.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:45.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:45.875+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:46.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:46.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:46.390+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:46.391+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:47.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:46.874+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:47.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:46.874+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:47.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:46.874+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:47.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:46.876+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:47.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:46.827+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:47.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:46.827+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:47.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:46.827+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:47.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:46.828+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:47.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:47.175+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13047 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:47.418+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:47.418+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:47.418+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:47.419+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:48.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:47.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:47.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:47.787+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:47.788+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:47.877+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:47.877+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:48.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:47.877+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:48.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:47.878+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:48.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:48.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:48.389+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:48.389+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:49.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:48.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:49.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:48.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:49.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:48.832+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:49.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:48.833+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:49.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:48.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:49.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:48.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:49.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:48.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:49.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:48.837+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:49.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:49.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:49.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:49.378+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:49.379+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:50.153 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:49.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:50.153 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:49.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:50.153 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:49.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:50.153 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:49.827+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:50.154 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:49.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:50.154 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:49.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:50.154 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:49.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:50.154 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:49.812+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:50.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:50.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:50.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:50.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:50.331+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:51.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:50.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:50.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:50.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:50.826+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:51.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:50.843+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:50.843+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:50.843+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:50.844+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:51.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:51.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:51.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:51.312+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:51.312+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:52.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:51.833+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:51.833+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:51.833+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:51.834+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:52.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:51.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:52.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:51.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:51.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:51.855+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:52.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:52.176+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13141 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:52.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:52.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:52.282+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:52.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:52.283+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:52.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:52.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:52.872+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:52.872+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:52.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:52.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:52.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:53.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:52.830+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:53.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:53.270+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:53.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:53.270+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:53.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:53.270+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:53.576 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:53.270+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:54.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:53.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:53.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:53.864+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:53.864+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:53.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:53.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:54.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:53.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:54.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:53.848+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:54.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:54.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:54.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:54.253+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:54.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:54.253+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:55.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:54.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:54.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:54.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:54.899+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:54.871+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:54.871+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:55.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:54.871+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:55.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:54.872+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:55.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:55.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:55.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:55.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:55.253+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:56.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:55.911+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:55.911+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:55.911+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:55.912+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:55.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:56.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:55.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:56.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:55.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:56.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:55.825+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:56.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:56.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:56.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:56.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:56.228+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:56.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:56.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:56.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:56.884+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:57.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:56.831+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:57.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:56.831+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:57.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:56.831+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:57.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:56.831+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:57.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:42:57.177+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13237 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:42:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:57.215+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:57.215+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:57.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:57.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:57.217+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:58.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:57.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:57.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:57.933+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:57.934+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:57.866+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:58.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:57.866+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:58.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:57.866+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:58.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:57.867+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:58.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:58.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:58.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:58.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:58.199+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:59.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:58.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:58.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:58.912+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:42:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:58.913+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:59.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:58.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:59.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:58.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:59.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:58.824+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:42:59.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:58.824+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:42:59.550 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:59.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:59.550 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:59.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:59.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:59.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:42:59.551 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:42:59.185+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:00.144 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:59.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:00.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:59.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:00.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:59.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:00.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:42:59.871+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:00.145 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:59.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:00.145 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:59.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:00.146 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:59.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:00.146 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:42:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:42:59.803+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:00.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:00.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:00.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:00.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:00.144+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:01.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:00.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:01.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:00.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:01.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:00.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:01.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:00.869+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:01.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:00.819+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:01.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:00.819+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:01.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:00.819+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:01.170 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:00.819+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:01.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:01.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:01.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:01.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:01.169+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:01.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:01.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:01.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:02.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:01.871+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:02.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:01.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:01.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:01.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:01.860+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:02.622 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:02.180+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13332 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:02.622 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:02.180+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:02.622 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:02.180+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:02.622 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:02.180+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:02.622 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:02.181+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:02.896 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:02.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.897 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:02.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.897 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:02.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:02.898 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:02.842+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:02.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:02.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:02.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:03.171 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:02.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:03.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:03.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:03.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:03.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:04.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:03.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:04.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:03.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:03.925+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:03.926+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:03.826+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:03.826+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:03.826+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:04.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:03.827+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:04.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:04.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:04.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:04.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:04.852+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:04.852+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:04.852+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:04.853+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:04.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:04.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:04.913+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:04.914+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:05.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:05.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:05.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:05.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:05.203+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:05.944 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:05.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.945 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:05.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.945 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:05.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:05.945 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:05.880+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:06.203 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:05.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:06.203 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:05.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:06.203 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:05.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:06.204 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:05.944+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:06.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:06.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:06.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:06.201+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:06.202+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:07.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:06.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:06.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:06.906+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:06.906+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:06.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:06.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:06.903+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:07.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:06.904+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:07.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:07.180+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13427 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:07.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:07.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:07.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:07.210+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:08.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:07.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:07.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:07.859+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:07.860+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:07.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:07.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:07.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:07.933+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:08.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:08.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:08.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:08.228+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:08.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:08.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:08.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:08.890+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:09.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:08.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:09.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:08.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:09.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:08.841+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:09.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:08.842+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:09.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:09.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:09.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:09.270+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:10.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:09.917+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:10.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:09.917+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:10.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:09.917+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:10.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:09.917+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:10.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:09.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:10.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:09.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:10.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:09.802+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:10.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:09.803+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:10.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:10.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:10.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:10.244+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:10.244+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:10.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:10.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:10.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:11.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:10.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:11.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:10.762+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:11.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:10.762+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:11.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:10.762+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:11.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:10.764+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:11.255+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:11.255+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:11.255+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:11.256+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:12.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:11.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:11.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:11.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:11.906+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:11.777+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:11.777+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:11.777+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:11.777+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:12.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:12.182+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13522 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:12.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:12.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:12.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:12.253+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:12.760+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:12.760+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:12.760+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:12.761+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:13.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:12.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:13.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:12.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:13.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:12.931+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:13.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:12.932+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:13.577 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:13.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:13.577 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:13.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:13.577 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:13.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:13.577 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:13.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:13.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:13.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:13.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:13.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:13.772+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:14.288 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:13.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:14.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:13.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:14.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:13.949+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:14.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:13.950+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:14.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:14.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:14.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:14.287+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:14.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:14.288+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:15.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:14.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:14.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:14.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:14.917+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:15.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:14.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:15.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:14.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:15.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:14.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:15.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:14.811+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:15.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:15.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:15.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:15.323+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:15.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:15.324+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:16.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:15.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:15.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:15.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:15.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:15.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:16.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:15.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:16.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:15.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:16.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:15.789+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:16.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:16.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:16.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:16.321+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:16.322+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:16.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:16.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:16.863+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:16.864+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:16.768+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:16.768+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:16.768+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:16.769+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:17.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:17.183+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13619 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:17.285+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:17.285+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:17.285+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:17.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:17.286+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:18.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:17.857+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:17.857+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:17.857+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:17.858+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:17.773+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:18.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:17.773+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:17.773+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:17.773+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:18.271+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:18.271+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:18.271+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:18.272+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:18.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:18.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:18.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:19.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:18.886+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:18.797+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:18.797+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:18.797+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:18.798+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:19.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:19.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:19.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:19.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:19.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:20.154 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:19.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:19.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:19.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:19.920+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:19.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:19.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:20.155 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:19.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:20.156 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:19.763+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:20.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:20.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:20.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:20.250+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:21.054 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:20.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:21.054 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:20.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:21.054 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:20.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:21.054 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:20.801+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:21.055 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:20.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:21.055 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:20.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:21.055 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:20.905+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:21.055 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:20.905+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:21.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:21.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:21.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:21.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:21.269+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:21.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:21.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:21.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:21.876+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:22.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:21.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:22.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:21.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:22.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:21.829+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:22.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:21.830+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:22.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:22.185+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13714 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:22.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:22.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:22.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:22.263+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:22.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:22.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:22.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:22.847+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:23.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:22.839+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:23.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:22.839+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:23.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:22.839+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:23.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:22.840+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:23.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:23.245+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:23.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:23.245+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:23.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:23.245+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:23.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:23.246+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:23.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:23.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:23.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:23.818+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:23.840 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:23.819+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:24.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:23.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:24.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:23.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:24.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:23.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:24.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:23.839+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:24.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:24.231+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:24.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:24.231+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:24.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:24.231+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:24.828+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:24.828+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:24.828+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:24.829+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:25.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:24.814+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:24.814+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:24.814+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:24.815+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:25.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:25.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:25.239+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:25.240+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:25.824+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:25.824+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:25.824+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:25.825+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:26.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:25.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:26.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:25.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:26.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:25.800+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:26.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:25.800+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:26.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:26.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:26.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:26.263+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:27.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:26.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:26.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:26.874+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:26.875+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:27.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:26.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:27.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:26.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:27.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:26.761+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:27.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:26.762+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:27.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:27.186+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13808 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:27.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:27.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:27.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:27.228+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:28.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:27.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:27.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:27.788+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:27.789+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:27.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:27.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:27.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:27.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:28.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:28.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:28.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:28.278+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:29.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:28.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:28.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:28.893+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:28.894+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:28.804+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:28.804+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:29.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:28.804+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:29.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:28.805+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:29.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:29.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:29.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:29.289+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:30.155 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:29.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:30.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:29.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:30.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:29.944+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:30.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:29.945+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:30.156 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:29.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:30.156 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:29.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:30.157 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:29.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:30.157 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:29.770+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:30.332+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:30.332+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:30.332+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:30.332+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:31.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:30.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:30.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:30.956+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:30.957+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:31.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:30.749+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:30.749+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:30.749+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:30.750+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:31.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:31.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:31.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:31.319+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:31.991 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:31.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.992 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:31.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.992 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:31.756+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:31.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:31.756+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:32.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:32.187+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13902 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:32.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:31.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:32.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:31.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:32.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:31.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:32.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:31.991+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:32.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:32.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:32.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:32.314+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:32.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:32.315+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:33.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:32.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:32.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:32.750+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:32.750+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:33.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:33.578 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:33.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:33.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:33.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:33.579 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:33.297+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:33.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:33.774+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:33.774+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:33.774+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:33.775+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:34.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:34.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:34.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:34.296 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:33.987+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:34.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:34.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:34.294+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:34.295+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:35.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:34.794+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:35.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:34.794+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:35.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:34.794+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:35.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:34.794+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:34.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:34.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:34.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:34.978+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:35.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:35.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:35.278+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:35.278+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:36.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:35.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:35.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:35.769+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:35.770+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:35.976+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:35.976+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:35.976+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:36.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:35.977+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:36.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:36.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:36.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:36.305+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:36.987 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:36.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.987 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:36.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.987 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:36.726+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:36.987 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:36.727+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:37.315 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:37.189+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 13998 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:37.315 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:36.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:37.315 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:36.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:37.315 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:36.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:37.316 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:36.986+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:37.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:37.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:37.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:37.346+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:37.997 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:37.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:37.997 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:37.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:37.997 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:37.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:37.997 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:37.679+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:38.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:37.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:38.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:37.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:38.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:37.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:38.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:37.996+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:38.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:38.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:38.349+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:38.350+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:38.992 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:38.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:38.992 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:38.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:38.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:38.709+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:38.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:38.710+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:39.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:38.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:39.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:38.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:39.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:38.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:39.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:38.991+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:39.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:39.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:39.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:39.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:39.666 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:39.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:39.667 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:39.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:39.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:39.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:39.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:39.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:39.665+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:40.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:40.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:40.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:40.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:40.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:40.009+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:40.319 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:40.010+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:40.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:40.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:40.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:40.318+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:41.025 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:40.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:41.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:40.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:41.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:40.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:41.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:40.697+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:41.321 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:41.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:41.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:41.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:41.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:41.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:41.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:41.025+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:41.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:41.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:41.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:41.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:41.320+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:42.005 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:41.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:41.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:41.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.006 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:41.689+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:42.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:42.190+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14090 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:42.293 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:42.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:42.293 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:42.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:42.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:42.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:42.294 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:42.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:42.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:42.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:42.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:42.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:42.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:42.291+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:42.558 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:42.292+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:42.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:42.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:42.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:42.716+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:42.717+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:43.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:43.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:43.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:43.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:43.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:43.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:43.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:43.293+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:43.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:43.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:43.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:43.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:43.669+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:44.337 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:44.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:44.337 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:44.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:44.337 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:44.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:44.337 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:44.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:44.335+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:44.335+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:44.335+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:44.336+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:45.045 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:44.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:44.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:44.718+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.046 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:44.718+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:45.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:45.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:45.328 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:45.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:45.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:45.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:45.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:45.045+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:45.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:45.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:45.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:45.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:45.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:45.327+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:45.632 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:45.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:45.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:45.741+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:45.741+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:45.741+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:45.741+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:46.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:46.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:46.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:46.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:46.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:46.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:46.346 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:46.092+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:46.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:46.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:46.345+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:46.346+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:47.051 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:46.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.051 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:46.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.051 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:46.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.051 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:46.704+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:47.319 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:47.193+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14183 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:47.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:47.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:47.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:47.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:47.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:47.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:47.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:47.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:47.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:47.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:47.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:47.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:47.664 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:47.392+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:47.664 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:47.392+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:47.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:47.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:47.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:47.662+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:47.663+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:48.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:48.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:48.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:48.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:48.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:48.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:48.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:48.036+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:48.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:48.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:48.357+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:48.358+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:49.023 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:48.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.024 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:48.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.024 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:48.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.024 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:48.677+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:49.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:49.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:49.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:49.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:49.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:49.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:49.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:49.024+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:49.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:49.311+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:49.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:49.311+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:49.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:49.311+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:49.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:49.312+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:49.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:49.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:49.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:49.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:49.643+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:50.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:50.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:50.064+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:50.065+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:50.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:50.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:50.289+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:50.290+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:51.107 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:50.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.108 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:50.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.108 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:50.683+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.108 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:50.683+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:51.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:51.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:51.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:51.264+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:51.265+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:51.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:51.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:51.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:51.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:51.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:51.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:51.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:51.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:51.640+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:52.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:52.194+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14275 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:52.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:52.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:52.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:52.089+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:52.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:52.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:52.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:52.297+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:52.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:52.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:52.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:52.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:52.647+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:53.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:53.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:53.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:53.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:53.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:53.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:53.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:53.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:53.322+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:53.323+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:53.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:53.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:53.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:53.688+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:53.689+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:54.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:54.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:54.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:54.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:54.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:54.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:54.326+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:54.327+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:55.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:54.728+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:54.728+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:54.728+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:54.729+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:55.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:55.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:55.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:55.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:55.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:55.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:55.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:55.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:55.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:55.296+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:56.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:55.714+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:56.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:55.714+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:56.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:55.714+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:56.160 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:55.715+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:56.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:56.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:56.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:56.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:56.158+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:56.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:56.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:56.302+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:56.302+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:57.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:56.751+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:57.139 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:56.751+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:57.140 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:56.751+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:57.140 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:56.752+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:57.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:43:57.196+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14371 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:43:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:57.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:57.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:57.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:57.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:57.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:57.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:57.295+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:57.296+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:58.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:57.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:58.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:57.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:58.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:57.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:58.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:57.764+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:58.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:58.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:58.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:58.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:58.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:58.250+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:58.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:58.250+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:58.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:58.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:58.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:58.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:58.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:58.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:58.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:58.173+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:59.165 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:58.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:59.166 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:58.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:59.166 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:58.792+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:43:59.166 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:58.793+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:59.224+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:59.224+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:59.224+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:43:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:43:59.225+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:43:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:59.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:59.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:59.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:43:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:43:59.165+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:00.151 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:59.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:00.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:59.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:00.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:59.840+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:00.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:43:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:43:59.840+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:00.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:00.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:00.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:00.151+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:00.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:00.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:00.275+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:00.276+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:01.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:00.880+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:00.880+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:00.880+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:01.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:00.881+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:01.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:01.315+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:01.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:01.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:01.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:01.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:01.199+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:02.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:01.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:02.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:01.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:02.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:01.847+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:02.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:01.848+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:02.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:02.197+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14463 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:02.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:02.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:02.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:02.222+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:02.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:02.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:02.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:02.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:02.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:02.267+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:02.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:02.268+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:03.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:02.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:02.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:02.854+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:02.855+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:03.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:03.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:03.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:03.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:03.218+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:03.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:03.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:03.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:03.313+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:03.583 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:03.313+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:04.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:03.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:03.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:03.837+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:03.838+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:04.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:04.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:04.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:04.319+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:04.320+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:04.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:04.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:04.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:04.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:04.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:04.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:05.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:04.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:04.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:04.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:05.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:04.818+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:05.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:05.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:05.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:05.181+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:05.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:05.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:05.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:05.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:05.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:05.277+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:06.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:05.807+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:05.807+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:05.807+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:06.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:05.808+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:06.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:06.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:06.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:06.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:06.176+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:06.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:06.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:06.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:06.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:06.256+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:06.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:06.256+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:07.134 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:07.135 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:07.135 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:07.135 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:06.799+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:07.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:07.199+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14557 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:07.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:07.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:07.269+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:07.269+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:07.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:07.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:07.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:07.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:08.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:07.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:07.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:07.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:07.819+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:08.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:08.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:08.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:08.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:08.239+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:08.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:08.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:08.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:08.225+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:09.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:08.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:09.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:08.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:09.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:08.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:09.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:08.863+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:09.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:09.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:09.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:09.211+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:09.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:09.211+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:09.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:09.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:09.273+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:09.273+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:10.158 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:09.895+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:10.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:09.895+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:10.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:09.895+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:10.159 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:09.896+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:10.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:10.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:10.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:10.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:10.228+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:10.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:10.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:10.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:10.230+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:11.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:10.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:10.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:10.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:10.880+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:11.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:11.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:11.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:11.237+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:11.238+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:11.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:11.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:11.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:11.234+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:11.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:11.234+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:12.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:11.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:11.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:11.879+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:11.880+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:12.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:12.200+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14649 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:12.207+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:12.207+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:12.207+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:12.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:12.209+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:12.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:12.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:12.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:12.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:12.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:12.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:12.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:12.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:13.171 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:12.913+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:13.171 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:12.913+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:13.171 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:12.913+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:13.171 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:12.914+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:13.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:13.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:13.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:13.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:13.170+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:13.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:13.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:13.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:13.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:14.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:14.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:14.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:14.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:14.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:14.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:14.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:14.148+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:14.244 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:13.938+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:14.244 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:13.938+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:14.244 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:13.938+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:14.245 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:13.939+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:14.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:14.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:14.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:14.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:14.242+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:15.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:14.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:14.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:14.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:14.961+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:15.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:15.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:15.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:15.148+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:15.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:15.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:15.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:15.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:15.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:15.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:15.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:16.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:16.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:16.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:16.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:16.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:16.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:16.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:16.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:16.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:15.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:16.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:15.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:16.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:15.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:16.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:15.990+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:16.351 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:16.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:16.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:16.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:16.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:16.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:16.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:16.124+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:17.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:17.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:17.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:17.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:17.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:17.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:17.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:17.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:17.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:17.000+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:17.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:17.000+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:17.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:17.000+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:17.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:17.000+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:17.331 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:17.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:17.331 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:17.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:17.331 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:17.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:17.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:17.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:17.332 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:17.202+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14745 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:18.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:18.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:18.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:18.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:18.122+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:18.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:18.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:18.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:18.033+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:18.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:18.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:18.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:18.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:18.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:18.155+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:19.143+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:19.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:19.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:19.144+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:19.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:19.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:19.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:19.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.023+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.023+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.023+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.024+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.979+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.979+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:20.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.979+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:20.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:19.980+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:20.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:20.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:20.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:20.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:20.127+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:20.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:20.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:20.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:20.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:20.148+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:21.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:21.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:21.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:21.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:21.114+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:20.975+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:20.975+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:20.975+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:20.976+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:21.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:21.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:21.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:21.134+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:22.099 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:21.935+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:22.099 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:21.935+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:22.099 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:21.935+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:22.099 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:21.936+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:22.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:22.203+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14837 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:22.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:22.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:22.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:22.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:22.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:22.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:22.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:22.098+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:23.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:23.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:23.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:23.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:23.134+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:22.939+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:22.939+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:22.939+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:22.940+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:23.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:23.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:23.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:23.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:23.113+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:24.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:24.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:24.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:24.122+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:23.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:23.954+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:23.954+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:23.954+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:24.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:24.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:24.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:24.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:24.126+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:25.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:25.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:25.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:25.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:25.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:25.120+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:25.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:25.120+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:25.340 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:24.959+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:25.341 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:24.959+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:25.341 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:24.959+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:25.341 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:24.960+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:25.341 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:25.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:25.341 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:25.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:25.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:25.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:25.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:25.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-19T06:44:25.343 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-01-19T06:44:25.343 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:44:25.344 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:44:25.360 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-19T06:44:25.360 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:44:25.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T06:44:26.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:26.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:26.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:26.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:26.104+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:26.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:26.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:26.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:26.214+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:25.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:25.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:26.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:25.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:26.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:25.997+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:27.215 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:26.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:27.215 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:26.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:27.215 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:26.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:27.215 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:26.957+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:27.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:27.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:27.216 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:27.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:27.216 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:27.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:27.216 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:27.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:27.216 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:27.205+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 14931 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:27.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:27.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:27.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:27.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:27.241+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:28.283 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:27.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:28.283 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:27.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:27.951+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:27.952+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:28.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:28.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:28.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:28.284 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:28.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:28.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:28.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:28.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:28.281+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:28.282+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:29.234 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:29.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:29.235 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:29.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:29.235 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:29.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:29.235 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:29.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:29.235 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:28.947+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:29.235 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:28.947+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:29.236 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:28.947+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:29.236 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:28.948+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:29.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:29.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:29.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:29.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:29.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:29.234+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:30.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:30.209+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:30.209+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:30.209+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:30.209+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:30.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:30.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:30.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:30.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:29.961+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:29.961+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:29.961+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:29.962+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:31.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:31.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:31.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:31.111+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:30.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:30.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:30.989+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:30.990+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:31.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:31.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:31.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:31.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:31.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:31.183+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:32.333 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:32.208+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15024 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:32.333 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:32.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:32.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:32.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:32.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:31.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:31.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:32.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:31.985+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:32.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:31.986+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:32.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:32.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:32.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:32.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:32.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:32.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:32.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:32.161+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:33.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:32.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:32.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:32.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:32.954+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:33.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:33.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:33.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:33.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:33.154+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:34.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:33.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:34.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:33.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:34.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:33.909+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:34.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:33.910+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:34.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.993+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:34.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:33.994+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:34.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:34.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:34.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:34.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:34.195+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:35.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:34.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:34.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:34.988+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:34.989+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:35.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:34.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:34.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:34.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:35.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:34.957+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:35.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:35.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:35.236+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:35.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:35.237+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:36.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:35.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:35.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:35.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.279 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:35.985+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:36.279 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:35.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:35.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:35.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:35.984+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:36.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:36.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:36.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:36.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:36.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:36.277+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:36.997 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:36.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.997 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:36.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.997 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:36.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:36.997 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:36.978+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:36.997 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:36.993+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.998 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:36.993+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.998 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:36.993+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:36.998 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:36.994+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:37.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:37.209+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15121 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:37.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:37.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:37.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:37.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:37.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:37.240+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.009+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:37.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:37.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:37.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:37.942+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:38.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:38.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:38.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:38.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:38.281+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:38.984 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:38.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.985 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:38.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.985 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:38.936+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:38.985 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:38.937+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:39.249 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:39.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:39.249 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:39.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:39.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:39.250 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:38.984+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:39.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:39.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:39.247+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:39.248+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:40.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:39.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:40.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:39.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:40.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:39.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:40.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:39.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:40.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:39.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:40.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:39.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:40.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:39.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:40.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:39.980+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:40.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:40.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:40.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:40.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:40.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:40.217+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:41.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:40.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:40.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:40.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:40.902+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:41.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:40.957+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:41.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:40.957+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:41.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:40.957+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:41.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:40.958+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:41.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:41.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:41.238+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:41.239+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:41.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:41.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:41.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:41.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:41.958 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:41.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:42.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:41.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:42.213 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:41.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:42.213 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:41.956+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:42.213 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:41.957+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:42.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:42.210+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15214 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:42.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:42.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:42.240+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:42.241+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:42.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:42.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:42.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:43.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:42.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:42.994+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:42.994+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:42.994+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:42.994+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:43.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:43.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:43.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:43.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:43.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:43.290+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:43.584 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:43.291+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:44.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:43.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:43.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:43.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:43.871+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:44.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:43.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:43.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:43.980+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:44.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:43.981+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:44.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:44.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:44.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:44.277+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:44.278+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:44.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:44.889+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:44.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:44.890+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:45.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:44.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:45.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:44.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:45.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:44.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:45.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:44.996+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:45.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:45.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:45.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:45.252+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:45.253+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:46.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:45.984+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:45.984+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:45.984+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:45.985+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:45.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:45.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:45.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:45.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:46.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:46.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:46.230+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:46.230+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:47.217 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:46.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:47.217 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:46.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:47.218 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:46.936+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:47.218 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:46.936+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:47.219 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:46.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:47.219 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:46.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:47.219 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:46.932+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:47.219 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:46.933+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:47.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:47.212+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15309 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:47.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:47.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:47.225+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:47.226+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:47.930+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:47.930+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:47.930+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:47.930+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:48.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:47.910+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:48.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:47.910+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:48.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:47.910+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:48.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:47.910+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:48.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:48.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:48.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:48.232+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:48.233+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:49.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:49.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:49.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:49.197+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:48.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:48.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:48.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:48.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:48.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:48.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:48.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:49.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:48.961+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:50.207 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:49.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:49.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:49.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:49.993+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:49.925+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:49.925+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:50.208 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:49.925+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:50.209 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:49.925+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:50.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:50.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:50.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:50.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:50.206+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:51.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:51.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:51.210+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:51.211+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:50.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:50.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:51.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:50.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:51.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:50.970+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:52.216 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:51.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:52.216 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:51.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:52.216 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:51.960+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:52.216 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:51.961+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:51.998+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:52.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:52.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:52.217 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:52.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:52.218 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:52.179+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:52.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:52.214+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15402 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:53.214 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:52.962+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:53.214 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:52.962+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:53.214 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:52.962+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:53.214 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:52.962+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:53.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:52.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:53.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:52.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:53.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:52.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:53.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:52.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:53.215 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:53.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:53.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:53.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:53.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:53.131+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:53.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:53.132+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:53.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:53.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:53.985+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:53.986+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:54.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:54.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:54.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:54.002+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:54.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:54.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:54.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:54.155+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:55.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:55.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:55.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:55.144+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:54.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:54.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:54.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:54.974+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:55.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:55.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:55.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:55.033+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:56.286 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:56.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:56.286 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:56.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:56.286 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:56.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:56.286 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:56.167+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:56.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:55.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:56.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:55.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:56.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:55.957+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:56.287 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:55.958+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:56.287 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:56.019+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:56.288 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:56.019+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:56.288 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:56.019+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:56.288 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:56.020+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:57.217 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:56.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:57.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:56.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:57.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:56.948+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:57.218 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:56.948+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:57.218 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:57.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:57.218 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:57.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:57.005+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:57.006+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:57.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:57.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:57.158+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:57.219 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:57.158+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:57.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:44:57.215+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15499 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:44:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:57.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:57.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:57.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:57.999+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:58.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:58.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:58.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:58.017+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:58.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:58.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:58.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:58.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:58.124+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:59.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:59.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:59.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:59.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:44:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:44:59.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:59.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:59.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:59.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:44:59.169+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:59.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:59.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:59.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:44:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:44:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:44:59.037+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:00.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:00.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:00.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:00.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:00.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:00.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:00.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:00.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:00.040+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:00.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:00.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:00.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:00.196+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:01.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:01.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:01.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:01.075+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:01.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:01.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:01.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:01.069+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:01.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:01.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:01.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:01.170+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:02.343 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:02.217+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15592 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:02.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:02.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:02.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:02.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:02.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:02.036+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:02.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:02.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:02.344 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:02.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:02.344 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:02.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:02.344 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:02.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:02.344 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:02.186+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:02.345 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:02.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:02.345 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:02.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:02.345 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:02.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:02.345 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:02.030+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:03.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:03.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:03.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:03.020+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:03.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:03.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:03.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:03.173+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:03.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:03.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:03.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:03.037+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:04.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:04.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:04.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:04.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:04.186+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:04.065+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:04.065+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:04.065+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:04.066+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:04.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:04.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:04.992+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:05.043+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:05.043+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:05.043+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:05.044+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:05.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:05.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:05.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:05.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:05.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:05.154+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.027+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:06.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:06.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:06.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:06.122+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:06.025+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:06.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:06.025+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:06.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:06.025+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:06.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:06.026+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:07.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:07.219+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15687 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:07.244 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.023+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:07.090+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:07.090+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:07.245 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:07.090+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:07.246 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:07.091+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:07.246 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:07.246 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:07.246 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:07.246 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:06.984+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:08.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:08.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:08.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:08.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:08.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.974+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.974+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.974+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:07.975+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:08.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:08.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:08.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:08.090+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:09.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:09.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:09.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:09.041+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:08.981+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:08.981+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:08.981+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:08.982+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:09.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:09.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:09.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:09.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:09.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:09.120+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:10.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:10.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:10.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:10.031+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:10.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:10.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:10.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:10.007+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:10.105+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:10.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:10.105+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:10.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:10.105+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:10.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:10.105+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:11.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:11.059+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:11.059+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:11.059+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:11.060+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:11.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:11.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:11.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:11.128+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:11.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:11.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:11.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:11.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:12.347 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:12.220+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15780 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:12.347 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:12.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:12.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:12.029+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:12.029+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:12.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:12.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:12.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:12.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:12.349 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:12.087+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:12.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:12.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:12.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:12.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:12.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:12.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:12.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:12.160+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:13.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:13.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:13.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:13.111+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:13.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:13.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:13.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:13.184+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:13.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:13.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:13.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:13.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:13.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:14.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:14.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:14.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:14.218+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:14.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:14.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:14.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:14.070+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:14.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:14.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:14.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:14.061+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:15.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:15.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:15.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:15.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:15.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:15.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:15.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:15.077+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:15.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:15.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:15.194+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:15.194+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:16.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:16.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:16.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:16.174+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:16.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:16.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:16.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:16.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:16.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:16.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:16.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:16.073+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:17.347 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:17.223+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15877 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:17.347 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:17.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:17.347 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:17.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:17.347 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:17.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:17.348 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:17.040+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:17.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:17.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:17.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:17.348 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:17.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:17.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:17.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:17.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:17.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:17.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:17.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:17.349 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:17.184+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:18.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:18.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:18.032+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:18.033+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:18.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:18.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:18.016+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:18.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:18.016+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:18.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:18.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:18.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:18.182+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:18.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:18.183+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:19.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:19.007+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:19.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:19.007+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:19.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:19.007+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:19.008+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:19.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:19.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:19.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:19.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:19.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:19.060+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:19.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:19.061+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:19.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:19.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:19.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:19.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:19.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:19.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:19.179+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:20.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:20.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:20.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:20.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:20.145+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:20.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:20.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:20.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:20.048+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:20.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:20.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:20.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:20.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:20.036+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:21.165 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:21.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:21.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:21.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:21.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:21.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:21.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:21.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:21.166 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:21.034+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:21.166 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:21.034+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:21.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:21.034+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:21.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:21.035+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:21.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:21.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:21.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:21.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:21.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:22.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:22.224+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 15970 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:22.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:22.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:22.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:22.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:22.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:22.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:22.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:22.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:22.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:22.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:22.023+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:22.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:22.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:22.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:22.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:22.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:22.165+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:22.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:22.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:23.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:23.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:23.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:23.185+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:23.186+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:23.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:23.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:23.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:23.027+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:23.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:23.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:23.018+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:23.018+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:24.219 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:24.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:24.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:24.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:24.011+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:24.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:24.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:24.220 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:24.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:24.221 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:24.040+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:24.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:24.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:24.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:24.217+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:24.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:24.218+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:25.262 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:25.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:25.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:25.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:25.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:25.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:25.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:25.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:25.263 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:25.089+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:25.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:25.260+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:25.260+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:25.260+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:25.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:25.261+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:26.306 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:26.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:26.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:26.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.012+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:26.307 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:26.308 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:26.308 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.039+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:26.308 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.039+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:26.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:26.305+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:26.305+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:26.305+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:26.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:26.305+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:27.237 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.992+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:27.237 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.992+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:27.237 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.992+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:27.237 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:26.993+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:27.225+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16065 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:27.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:27.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:27.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:27.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:26.981+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:27.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:27.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:27.257+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:27.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:27.258+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:28.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:28.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:28.021+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:28.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:28.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:28.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:28.262+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:28.263+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.033+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.034+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:28.987+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:29.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:29.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:29.226+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:29.227+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:29.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:29.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:29.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:29.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:29.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:29.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:30.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:30.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:30.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:30.038+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:30.038+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:30.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:30.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:30.213+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:30.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:30.214+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:31.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:31.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:31.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:31.071+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:31.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:31.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:31.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:31.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:31.088+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:31.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:31.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:31.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:31.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:31.193+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:32.353 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:32.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:32.353 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:32.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:32.353 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:32.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:32.353 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:32.049+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:32.354 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:32.115+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:32.354 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:32.115+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:32.354 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:32.115+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:32.354 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:32.115+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:32.355 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:32.243+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:32.355 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:32.243+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:32.355 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:32.243+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:32.355 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:32.244+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:32.355 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:32.227+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16158 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:33.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:33.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:33.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:33.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:33.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:33.227+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:33.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:33.227+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:33.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:33.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:33.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:33.042+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:33.081+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:33.081+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:33.081+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:33.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:33.082+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:34.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:34.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:34.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:34.089+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:34.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:34.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:34.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:34.197+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:34.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:34.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:34.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:34.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:34.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:35.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:35.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:35.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:35.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:35.093+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:35.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:35.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:35.040+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:35.041+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:35.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:35.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:35.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:35.187+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:36.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:36.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:36.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:36.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:36.135+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:36.069+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:36.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:36.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:36.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:36.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:36.166+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:37.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:37.229+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16255 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:37.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:37.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:37.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:37.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:37.119+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:37.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:37.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:37.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:37.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:37.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:37.162+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:37.353 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:37.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:37.353 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:37.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:37.353 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:37.096+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:37.353 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:37.098+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:38.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:38.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:38.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:38.103+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:38.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:38.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:38.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:38.194+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:38.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:38.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:38.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:38.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:38.069+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:39.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:39.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:39.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:39.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:39.093+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:39.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:39.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:39.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:39.064+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:39.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:39.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:39.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:39.220+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:40.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:40.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:40.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:40.099+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:40.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:40.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:40.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:40.052+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:40.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:40.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:40.205+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:40.205+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:41.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:41.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:41.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:41.142+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:41.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:41.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:41.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:41.102+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:41.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:41.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:41.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:41.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:41.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:41.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:41.192+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:42.356 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:42.231+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16348 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:42.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:42.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:42.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:42.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:42.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:42.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:42.357 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:42.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:42.358 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:42.060+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:42.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:42.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:42.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:42.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:42.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:42.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:42.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:42.203+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:43.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:43.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:43.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:43.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:43.187+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:43.109+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:43.109+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:43.109+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:43.110+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:43.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:43.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:43.216+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:43.216+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:44.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:44.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:44.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:44.199+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:44.061+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:44.061+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:44.061+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:44.061+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:44.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:44.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:44.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:44.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:44.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:44.235+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:44.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:44.236+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:45.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:45.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:45.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:45.022+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:45.156+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:45.156+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:45.156+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:45.157+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:45.233+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:45.233+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:45.233+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:45.234+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:46.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:46.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:46.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:46.203+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:46.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:46.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:46.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:46.114+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:46.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:46.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:46.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:46.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:46.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:46.012+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:47.358 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:47.232+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16441 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:47.359 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:47.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:47.359 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:47.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:47.359 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:47.032+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:47.359 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:47.033+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:47.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:47.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:47.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:47.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:47.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:47.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:47.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:47.175+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:47.361 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:47.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:47.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:47.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:47.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:47.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:47.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:47.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:48.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:48.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:48.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:48.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:48.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:48.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:48.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:48.063+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:48.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:48.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:48.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:48.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:48.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:49.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:49.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:49.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:49.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:49.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:49.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:49.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:49.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:49.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:49.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:49.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:49.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:49.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:49.085+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:50.151+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:50.151+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:50.151+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:50.152+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:50.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:50.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:50.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:50.072+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:50.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:50.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:50.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:50.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:50.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:50.164+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:51.114+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:51.114+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:51.114+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:51.115+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:51.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:51.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:51.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:51.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:51.049+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:51.049+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:51.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:51.049+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:51.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:51.050+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:52.359 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:52.234+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16533 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:52.360 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:52.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:52.360 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:52.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:52.361 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:52.069+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:52.361 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:52.070+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:52.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:52.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:52.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:52.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:52.362 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:52.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:52.363 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:52.112+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:52.363 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:52.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:52.363 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:52.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:52.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:52.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:52.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:52.182+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:53.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:53.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:53.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:53.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:53.056+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:53.056+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:53.056+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:53.057+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:53.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:53.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:53.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:53.122+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:54.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:54.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:54.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:54.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:54.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:54.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:54.062+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:54.063+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:54.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:54.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:54.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:54.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:55.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:55.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:55.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:55.168+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:55.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:55.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:55.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:55.234+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:55.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:55.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:55.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:55.037+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:56.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.011+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.011+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:56.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:56.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:56.155+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:56.157+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:57.238 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:57.238 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:57.238 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.970+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:57.238 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:56.971+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:57.239 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:57.239 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:57.239 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:57.239 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:57.144+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:57.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:45:57.236+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16629 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:45:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:57.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:57.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:57.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:57.290+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:58.254 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:57.999+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:58.254 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:57.999+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:58.254 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:57.999+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:58.254 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:58.000+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:58.255 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:58.255 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:58.255 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:58.255 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:58.194+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:58.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:58.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:58.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:58.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:58.253+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:59.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:58.958+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:58.958+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:58.958+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:58.959+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:59.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:59.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:59.203+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:45:59.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:45:59.204+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:45:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:59.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:59.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:59.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:45:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:45:59.269+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:00.279 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:59.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:59.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:59.983+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:45:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:45:59.983+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:00.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:00.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:00.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:00.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:00.281 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:00.191+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:00.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:00.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:00.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:00.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:00.278+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:01.173 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:00.987+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:01.173 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:00.987+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:01.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:00.987+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:01.174 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:00.987+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:01.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:01.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:01.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:01.231+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:01.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:01.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:01.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:01.173+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:02.263 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:02.239+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16721 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:02.264 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:02.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:02.264 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:02.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:02.264 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:02.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:02.264 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:02.225+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:02.265 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:02.009+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:02.265 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:02.009+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:02.265 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:02.009+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:02.265 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:02.010+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:02.265 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:02.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:02.266 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:02.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:02.266 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:02.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:02.266 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:02.161+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:03.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:03.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:03.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:03.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:03.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:03.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:03.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:03.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:03.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:03.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:03.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.036+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:04.162 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.990+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:04.162 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.990+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:04.162 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.990+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:04.163 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:03.990+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:04.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:04.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:04.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:04.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:04.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:04.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:04.160+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:04.161+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:05.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:05.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:05.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:05.198+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:04.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:04.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:04.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:04.996+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:05.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:05.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:05.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:05.180+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:06.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:06.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:06.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:06.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:06.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:06.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:06.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:06.029+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:06.030+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:06.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:07.197 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:07.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:07.198 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:07.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:07.198 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:07.125+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:07.198 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:07.126+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:07.198 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:07.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:07.198 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:07.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:07.199 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:07.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:07.199 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:07.027+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:07.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:07.239+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16816 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:07.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:07.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:07.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:07.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:08.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:08.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:08.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:08.022+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:08.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:08.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:08.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:08.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:08.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:08.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:08.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:08.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:08.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:09.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:09.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:09.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:09.035+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:09.036+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:09.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:09.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:09.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:09.136+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:09.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:09.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:09.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:09.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:09.232+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:10.276 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:10.276 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:10.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:10.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:10.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:10.074+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:10.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:10.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:10.278 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:10.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:10.278 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:10.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:10.278 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:10.116+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:10.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:10.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:10.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:10.274+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:11.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:11.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:11.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:11.088+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:11.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:11.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:11.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:11.140+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:11.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:11.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:11.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:11.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:11.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:11.228+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:12.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:12.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:12.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:12.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:12.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:12.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:12.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:12.130+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:12.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:12.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:12.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:12.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:12.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:12.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:12.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:12.239+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:12.368 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:12.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:12.369 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:12.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:12.369 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:12.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:12.369 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:12.106+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:12.369 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:12.241+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 16909 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:13.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:13.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:13.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:13.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:13.099+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:13.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:13.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:13.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:13.087+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:13.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:13.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:13.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:13.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:13.277+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:14.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:14.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:14.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:14.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:14.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:14.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:14.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:14.051+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:14.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:14.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:14.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:14.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:14.255+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:15.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:15.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:15.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:15.231+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:15.083+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:15.083+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:15.083+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:15.084+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:15.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:15.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:15.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:15.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:16.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:16.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:16.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:16.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:16.194+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:16.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:16.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:16.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:16.131+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:16.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:16.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:16.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:16.062+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:17.372 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:17.243+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17000 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:17.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:17.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:17.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:17.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:17.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:17.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:17.373 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:17.233+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:17.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:17.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:17.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:17.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:17.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:17.131+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:17.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:17.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:17.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:17.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:17.060+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:18.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:18.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:18.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:18.215+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:18.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:18.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:18.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:18.050+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:18.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:18.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:18.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:18.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:18.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:18.114+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:19.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:19.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:19.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:19.255+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:19.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:19.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:19.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:19.014+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:19.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:19.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:19.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:19.160+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:20.287 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:20.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:20.287 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:20.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:20.287 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:20.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:20.287 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:20.188+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:20.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:20.004+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:20.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:20.004+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:20.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:20.004+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:20.288 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:20.004+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:20.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:20.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:20.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:20.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:20.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:21.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:21.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:21.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:21.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:21.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:21.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:21.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:21.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:21.032+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:21.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:21.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:21.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:21.229+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:22.259 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:22.245+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17091 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:22.259 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:22.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:22.259 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:22.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:22.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:22.236+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:22.005+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:22.005+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:22.005+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:22.260 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:22.006+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:22.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:22.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:22.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:22.264+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:23.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:23.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:23.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:23.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:23.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:23.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:23.267 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:23.244+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:23.267 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:23.002+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:23.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:23.002+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:23.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:23.002+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:23.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:23.002+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:23.592 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:23.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:23.592 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:23.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:23.592 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:23.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:23.592 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:23.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:24.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:24.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:24.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:24.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:24.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:24.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:24.302 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:24.048+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:24.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:24.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:24.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:24.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:24.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:24.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:24.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:24.267+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:24.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:24.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:24.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:24.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:24.301+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:25.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:25.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:25.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:25.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:25.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:25.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:25.298 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:25.293+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:25.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:25.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:25.295+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:25.296+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:25.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:25.299 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:25.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:25.300 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:25.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:25.300 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:25.046+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-19T06:46:25.490 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-01-19T06:46:25.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:46:25.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/clock: Cannot utime: Input/output error 2024-01-19T06:46:25.551 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy: Cannot mkdir: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_destroy/cln.sh: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init: Cannot mkdir: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.552 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_init/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock: Cannot mkdir: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/6-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.553 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/7-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedrdlock/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.554 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstack: Cannot mkdir: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstack/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstack/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstack/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getstack/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal: Cannot mkdir: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/testfrmw.c: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/testfrmw.h: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_equal/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy: Cannot mkdir: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_destroy/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink: Cannot mkdir: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/9-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/10-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.557 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/10-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/8-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.558 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/7-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/shm_unlink/11-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold: Cannot mkdir: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/3-2.sh: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/3-core-buildonly.c: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/3-4.sh: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sighold/3-3.sh: Cannot open: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock: Cannot mkdir: Input/output error 2024-01-19T06:46:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/6-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/7-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_rwlock_timedwrlock/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create: Cannot mkdir: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/speculative: Cannot mkdir: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_key_create/speculative/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock: Cannot mkdir: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.563 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_getclock/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach: Cannot mkdir: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/4-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/testfrmw.c: Cannot open: Input/output error 2024-01-19T06:46:25.564 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/testfrmw.h: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/4-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_detach/threads_scenarii.c: Cannot open: Input/output error 2024-01-19T06:46:25.565 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock: Cannot mkdir: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_spin_lock/3-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.566 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getschedparam: Cannot mkdir: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getschedparam/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getschedparam/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getschedparam/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_attr_getschedparam/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared: Cannot mkdir: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.567 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_setpshared/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime: Cannot mkdir: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.568 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/5-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.569 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/1-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/1-4.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/speculative: Cannot mkdir: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/speculative/6-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/speculative/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_gettime/speculative/6-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy: Cannot mkdir: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.570 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/testfrmw.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/testfrmw.h: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/5-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.571 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/speculative: Cannot mkdir: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutex_destroy/speculative/4-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue: Cannot mkdir: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/9-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/10-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/8-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/12-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/7-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigqueue/11-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific: Cannot mkdir: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_setspecific/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create: Cannot mkdir: Input/output error 2024-01-19T06:46:25.574 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-6.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/10-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/testfrmw.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/testfrmw.h: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/14-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.575 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/5-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/15-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/13.sh: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/8-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.576 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/12-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-5.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/8-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/1-4.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/threads_scenarii.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/11-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_create/3-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.577 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo: Cannot mkdir: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/9-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.578 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/8-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigwaitinfo/7-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getprotocol: Cannot mkdir: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getprotocol/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getprotocol/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getprotocol/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_mutexattr_getprotocol/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init: Cannot mkdir: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/4-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrier_init/2-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill: Cannot mkdir: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/kill/2-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime: Cannot mkdir: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/8-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.581 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/9-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/5-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/9-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/8-4.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.582 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/6-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/5-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/8-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/13-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/10-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/8-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.583 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/11-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/3-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/speculative: Cannot mkdir: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/speculative/12-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/speculative/12-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/speculative/12-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/timer_settime/3-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init: Cannot mkdir: Input/output error 2024-01-19T06:46:25.584 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_barrierattr_init/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min: Cannot mkdir: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.585 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/3-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/1-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sched_get_priority_min/1-4.c: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock: Cannot mkdir: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/1-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.586 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/1-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/pthread_condattr_setclock/1-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset: Cannot mkdir: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/4-2.sh: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/1-2.sh: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:gzip: tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/1-core-buildonly.c: Cannot open: Input/output error 2024-01-19T06:46:25.587 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:stdin: Stale file handle 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/4-core-buildonly.c: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/4-3.sh: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/1-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/assertions.xml: Cannot open: Input/output error 2024-01-19T06:46:25.588 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/4-1.sh: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/1-3.c: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/1-4.c: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sigdelset/4-4.sh: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink: Cannot mkdir: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/3-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/9-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/4-2.c: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/coverage.txt: Cannot open: Input/output error 2024-01-19T06:46:25.589 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/testfrmw.c: Cannot open: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/testfrmw.h: Cannot open: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/2-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces/sem_unlink/5-1.c: Cannot open: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: Unexpected EOF in archive 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance/interfaces: Cannot utime: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite/conformance: Cannot utime: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases/open_posix_testsuite: Cannot utime: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-01-19T06:46:25.590 INFO:tasks.workunit.client.0.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-01-19T06:46:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-19T06:46:25.592 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-01-19T06:46:25.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-19T06:46:25.592 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi134.front.sepia.ceph.com 2024-01-19T06:46:25.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T06:46:25.593 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-19T06:46:25.663 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-01-19T06:46:25.663 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:26.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:26.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:26.296+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:26.297+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:26.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:26.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:26.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:26.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:26.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:26.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:26.058+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:26.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:26.333+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:26.333+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:26.333+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:26.333+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:27.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:27.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:27.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:27.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:27.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:27.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:27.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:27.263+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:27.351 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:27.085+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:27.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:27.085+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:27.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:27.085+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:27.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:27.086+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:27.352 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:27.247+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17185 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:27.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:27.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:27.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:27.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:27.349+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:28.343+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:28.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:28.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:28.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:28.127+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:28.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:28.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:28.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:28.288+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:29.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:29.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:29.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:29.242+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:29.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:29.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:29.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:29.379+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:29.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:29.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:29.139+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:29.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:30.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:30.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:30.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:30.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:30.200+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:30.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:30.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:30.167+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:30.168+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:30.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:30.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:30.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:30.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:30.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:30.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:30.409+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:31.441 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:31.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:31.441 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:31.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:31.441 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:31.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:31.441 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:31.245+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:31.442 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:31.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:31.442 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:31.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:31.442 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:31.188+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:31.442 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:31.188+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:31.699 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:31.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:31.699 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:31.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:31.699 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:31.439+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:31.700 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:31.440+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:32.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:32.249+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17277 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:32.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:32.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:32.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:32.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:32.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:32.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:32.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:32.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:32.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:32.409+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:32.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:32.410+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:32.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:32.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:32.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:32.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:32.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:32.179+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:32.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:32.181+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:33.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:33.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:33.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:33.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:33.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:33.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:33.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:33.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:33.166+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:33.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:33.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:33.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:33.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:33.427+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:34.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:34.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:34.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:34.146+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:34.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:34.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:34.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:34.192+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:34.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:34.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:34.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:34.466+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:34.467+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:35.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:35.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:35.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:35.197+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:35.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:35.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:35.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:35.162+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:35.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:35.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:35.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:35.475+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:36.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:36.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:36.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:36.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:36.436+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:36.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:36.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:36.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:36.192+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:36.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:36.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:36.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:36.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:36.208+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:37.465 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:37.251+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17374 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:37.466 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:37.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:37.466 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:37.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:37.466 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:37.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:37.467 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:37.204+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:37.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:37.223+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:37.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:37.223+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:37.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:37.223+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:37.467 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:37.224+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:37.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:37.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:37.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:37.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:37.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:37.465+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:38.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:38.430+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:38.430+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:38.430+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:38.430+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:38.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:38.221+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:38.221+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:38.221+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:38.222+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:38.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:38.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:38.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:38.228+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:38.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:38.229+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:39.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:39.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:39.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:39.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:39.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:39.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:39.172+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:39.173+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:39.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:39.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:39.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:39.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:39.397+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:40.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:40.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:40.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:40.144+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:40.144+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:40.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:40.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:40.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:40.391+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:40.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:40.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:40.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:40.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:40.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:41.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:41.148+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:41.148+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:41.148+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:41.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:41.149+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:41.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:41.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:41.361+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:41.362+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:41.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:41.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:41.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:41.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:42.378 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:42.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:42.378 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:42.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:42.378 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:42.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:42.378 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:42.216+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:42.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:42.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:42.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:42.326+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:42.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:42.379 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:42.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:42.380 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:42.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:42.380 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:42.120+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:42.380 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:42.252+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17467 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:43.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:43.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:43.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:43.123+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:43.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:43.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:43.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:43.319+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:43.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:43.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:43.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:43.182+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:43.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:43.183+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:44.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:44.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:44.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:44.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:44.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:44.205+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:44.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:44.206+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:44.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:44.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:44.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:44.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:44.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:44.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:44.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:44.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:44.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:44.112+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:45.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:45.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:45.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:45.085+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:45.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:45.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:45.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:45.271+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:45.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:45.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:45.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:45.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:45.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:45.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:45.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:45.187+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:46.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:46.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:46.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:46.122+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:46.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:46.123+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:46.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:46.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:46.248+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:46.249+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:46.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:46.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:46.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:46.170+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:47.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:47.255+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17563 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:47.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:47.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:47.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:47.270+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:47.149+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:47.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:47.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:47.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:47.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:47.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:47.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:47.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:47.135+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:48.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:48.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:48.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:48.167+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:48.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:48.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:48.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:48.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:48.150+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:48.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:48.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:48.245+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:48.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:48.246+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:49.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:49.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:49.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:49.229+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:49.125+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:49.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:49.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:49.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:49.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:49.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:49.178+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:49.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:49.179+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:50.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:50.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:50.087+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:50.087+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:50.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:50.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:50.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:50.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:50.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:50.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:50.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:50.202+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:50.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:50.203+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:51.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:51.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:51.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:51.174+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:51.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:51.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:51.051+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:51.051+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:51.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:51.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:51.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:51.201+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:52.383 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:52.256+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17656 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:52.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:52.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:52.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:52.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:52.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:52.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:52.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:52.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:52.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:52.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:52.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:52.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:52.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:52.089+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:52.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:52.090+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:52.385 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:52.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:52.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:52.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:52.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:52.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:52.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:52.159+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:53.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:53.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:53.076+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:53.077+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:53.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:53.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:53.166+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:53.167+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:53.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:53.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:53.186+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:53.187+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:54.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:54.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:54.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:54.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:54.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:54.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:54.071+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:54.071+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:54.183+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:54.184+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:54.184+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:54.184+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:55.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:55.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:55.088+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:55.089+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:55.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:55.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:55.191+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:55.192+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:55.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:55.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:55.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:55.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:55.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:55.151+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:56.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:56.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:56.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:56.111+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:56.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:56.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:56.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:56.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:56.192+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:56.192+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:56.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:56.192+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:56.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:56.193+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:57.384 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:46:57.258+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17752 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:46:57.384 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:57.074+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:57.384 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:57.074+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:57.384 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:57.074+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:57.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:57.075+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:57.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:57.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:57.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:57.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:57.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:57.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:57.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:57.147+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:57.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:57.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:57.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:57.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:57.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:57.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:57.386 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:57.191+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:58.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:58.184+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:58.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:58.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:58.161+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:58.162+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:58.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:58.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:58.107+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:58.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:58.107+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:58.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:58.107+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:59.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:59.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:59.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:46:59.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:46:59.196+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:59.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:59.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:59.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:46:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:46:59.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:46:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:59.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:59.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:59.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:59.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:46:59.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:46:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:46:59.122+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:00.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:00.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:00.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:00.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:00.102+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:00.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:00.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:00.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:00.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:00.201+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:00.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:00.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:00.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:00.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:01.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:01.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:01.198+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:01.199+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:01.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:01.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:01.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:01.190+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:01.107+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:01.107+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:01.107+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:01.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:01.108+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:02.386 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:02.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:02.386 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:02.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:02.387 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:02.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:02.387 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:02.095+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:02.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:02.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:02.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:02.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:02.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:02.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:02.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:02.142+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:02.388 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:02.189+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:02.388 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:02.189+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:02.388 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:02.189+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:02.388 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:02.189+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:02.389 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:02.260+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17845 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:03.100+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:03.100+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:03.100+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:03.101+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:03.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:03.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:03.178+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:03.178+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:03.146+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:03.146+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:03.146+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:03.147+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:04.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:04.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:04.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:04.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:04.134+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:04.134+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:04.134+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:04.135+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:04.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:04.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:04.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:04.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:04.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:05.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:05.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:05.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:05.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:05.175+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:05.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:05.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:05.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:05.159+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:05.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:05.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:05.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:05.174+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:06.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:06.177+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:06.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:06.177+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:06.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:06.177+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:06.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:06.178+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:06.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:06.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:06.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:06.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:06.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:06.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:06.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:06.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:06.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:06.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:06.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:06.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:06.205+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:07.388 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:07.262+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 17939 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:07.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:07.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:07.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:07.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:07.388 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:07.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:07.389 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:07.112+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:07.389 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:07.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:07.389 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:07.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:07.389 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:07.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:07.389 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:07.168+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:07.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:07.145+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:07.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:07.145+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:07.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:07.145+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:07.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:07.146+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:08.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:08.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:08.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:08.155+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:08.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:08.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:08.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:08.080+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:08.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:08.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:08.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:08.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:08.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:08.181+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:09.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:09.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:09.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:09.123+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:09.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:09.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:09.168+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:09.169+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:09.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:09.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:09.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:09.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:10.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:10.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:10.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:10.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:10.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:10.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:10.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:10.150+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:10.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:10.130+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:11.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:11.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:11.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:11.093+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:11.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:11.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:11.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:11.127+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:11.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:11.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:11.098+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:11.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:11.099+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:12.391 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:12.264+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18032 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:12.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:12.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:12.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:12.064+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:12.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:12.392 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:12.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:12.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:12.095+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:12.097+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:12.097+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:12.097+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:12.393 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:12.097+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:13.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:13.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:13.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:13.138+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:13.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:13.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:13.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:13.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:13.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:13.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:13.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:13.099+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:14.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:14.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:14.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:14.091+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:14.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:14.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:14.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:14.099+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:14.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:14.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:14.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:14.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:14.054+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:15.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:15.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:15.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:15.059+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:15.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:15.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:15.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:15.011+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:15.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:15.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:15.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:15.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.008+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.008+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.008+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.008+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.041+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:16.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:16.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:16.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:16.103+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:16.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:16.104+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:17.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.959+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:17.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.959+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:17.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.959+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:17.268 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:16.960+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:17.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:17.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:17.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:17.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:17.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:17.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:17.269 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:17.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:17.269 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:17.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:17.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.995+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:17.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:16.996+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:17.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:17.266+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18128 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:18.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:18.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:18.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:18.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.041+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.042+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:18.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:17.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:18.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:17.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:18.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:17.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:18.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:17.982+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:18.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:18.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:18.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:18.967+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:19.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:19.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:19.125+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:19.126+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:18.998+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.020+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.020+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:20.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:20.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:20.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:20.080+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:19.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:19.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:20.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:19.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:20.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:19.999+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:21.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:21.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:21.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.997+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:21.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:20.998+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:21.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:21.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:21.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:21.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.002+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:22.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:22.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:22.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:22.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:22.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:22.030+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:22.270 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:22.031+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:22.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:22.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:22.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:22.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:22.271 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:22.272 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:22.272 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:21.966+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:22.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:22.268+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18221 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:23.425 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:22.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:22.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:22.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:22.928+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:23.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:23.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:23.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:23.122+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.004+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.004+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:23.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:23.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:23.924+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:23.925+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:24.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:24.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:24.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:24.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.958+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.958+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.959+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:24.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:23.960+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:24.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:24.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:24.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:24.922+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:25.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:25.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:25.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:25.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:25.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:25.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:25.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:25.158+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:24.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:25.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:24.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:25.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:24.986+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:25.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:24.987+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:25.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:25.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:25.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:25.888+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:26.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:26.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:26.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:26.128+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:26.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:26.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:26.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:26.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:26.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:26.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:26.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:26.036+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:27.149 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:26.874+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:27.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:27.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:27.026+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:27.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:27.026+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:27.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:27.270+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18315 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:27.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:27.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:27.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:27.148+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:28.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:27.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:28.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:27.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:28.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:27.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:28.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:27.856+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:28.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:28.047+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:28.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:28.047+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:28.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:28.047+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:28.119 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:28.048+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:28.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:28.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:28.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:28.117+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:29.060 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:28.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:29.060 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:28.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:29.061 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:28.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:29.061 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:28.902+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:29.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:29.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:29.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:29.162+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:29.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:29.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:29.058+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:29.059+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:30.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:30.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:30.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:30.036+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:30.167 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:30.036+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:30.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:29.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:30.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:29.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:30.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:29.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:30.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:29.866+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:30.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:30.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:30.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:30.166+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:31.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:31.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:31.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:31.150+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:31.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:31.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:31.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:31.001+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:31.178 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:31.002+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:31.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:30.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:31.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:30.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:31.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:30.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:31.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:30.901+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:32.022 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:31.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:32.022 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:31.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:32.022 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:31.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:32.023 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:31.946+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:32.274 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:32.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:32.274 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:32.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:32.021+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:32.022+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:32.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:32.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:32.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:32.275 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:32.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:32.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:32.273+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18409 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:33.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:32.983+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:33.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:32.983+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:33.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:32.983+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:33.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:32.984+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:33.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:33.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:33.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:33.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:33.174+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:33.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:33.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:33.037+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:33.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:33.038+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:33.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:33.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:33.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:34.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:33.984+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:34.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:34.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:34.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:34.025+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:34.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:34.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:34.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:34.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:34.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:34.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:35.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:35.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:35.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:35.051+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:35.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:35.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:35.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:35.166+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:35.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:34.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:35.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:34.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:35.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:34.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:35.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:34.965+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:36.184 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:36.184 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:36.184 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:36.068+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:36.184 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:36.069+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:36.185 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:35.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:36.185 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:35.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:36.185 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:35.927+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:36.185 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:35.928+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:36.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:36.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:36.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:36.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:36.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:37.276 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:37.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:37.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:37.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:37.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:37.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:37.277 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:37.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:37.278 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:37.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:37.278 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:37.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:37.278 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:37.090+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:37.278 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:37.091+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:37.279 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:36.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:37.279 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:36.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:37.279 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:36.956+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:37.279 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:36.957+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:37.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:37.274+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18505 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:38.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:38.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:38.113+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:38.114+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:37.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:38.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:37.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:38.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:37.981+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:38.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:37.982+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:38.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:38.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:38.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:38.241+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:39.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:39.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:39.130+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:39.131+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:39.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:39.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:39.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:39.216+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:39.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:39.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:39.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:39.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:39.019+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:40.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:40.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:40.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:40.107+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:40.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:40.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:40.235+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:40.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:40.236+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:40.016+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:40.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:40.016+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:40.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:40.016+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:40.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:40.016+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:41.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:41.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:41.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:41.242+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:41.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:41.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:41.131+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:41.132+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:41.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:41.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:41.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:41.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:41.046+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:42.404 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:42.276+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18597 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:42.435 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:42.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:42.436 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:42.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:42.436 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:42.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:42.436 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:42.287+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:42.436 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:42.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:42.436 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:42.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:42.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:42.094+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:42.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:42.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:42.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:42.437 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:42.056+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:43.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:43.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:43.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:43.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:43.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:43.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:08.001454+0000 (oldest deadline 2024-01-19T06:37:28.001454+0000) 2024-01-19T06:47:43.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:43.137+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:43.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:43.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:43.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:43.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:43.061+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:43.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:43.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:43.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:43.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:43.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:43.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:43.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:43.324+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:44.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:44.110+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:44.371 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:44.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:44.371 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:44.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:44.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:44.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:44.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:44.058+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:44.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:44.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:44.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:44.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:44.370+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:45.362 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:45.124+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:45.363 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:45.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:45.363 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:45.018+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:45.363 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:45.019+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:45.363 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:45.019+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:45.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:45.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:45.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:45.361+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:46.388 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:46.141+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:46.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:46.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:46.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:46.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:46.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:46.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:46.389 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:46.021+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:46.386+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:46.386+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:46.386+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:46.387+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:47.280 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:47.115+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:47.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:47.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:47.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:47.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:47.280 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:47.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:47.281 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:47.027+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:47.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:47.278+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18690 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:47.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:47.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:47.355+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:47.355+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:48.401 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:48.161+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:48.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:48.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:48.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:48.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:48.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:48.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:48.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:48.034+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:48.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:48.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:48.400+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:48.400+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:49.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:49.173+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:49.364 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:49.364 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:49.364 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.017+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:49.365 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.017+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:49.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:49.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:49.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:49.363+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:50.360 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:50.211+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:50.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:50.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:50.360 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.974+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:50.361 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:49.975+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:50.358+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:50.358+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:50.358+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:50.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:50.359+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:51.390 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:51.163+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:51.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:50.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:51.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:50.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:51.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:50.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:51.390 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:50.965+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:51.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:51.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:51.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:51.398+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:52.284 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:52.177+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:52.284 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:52.284 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:52.285 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:52.285 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.001+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:52.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:52.280+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18782 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:52.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:52.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:52.397+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:52.397+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:53.409 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:53.143+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:53.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:53.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:53.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.965+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:20.325888+0000 (oldest deadline 2024-01-19T06:37:40.325888+0000) 2024-01-19T06:47:53.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:52.966+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:53.407+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:53.407+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:53.407+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:37:21.213569+0000 (oldest deadline 2024-01-19T06:37:41.213569+0000) 2024-01-19T06:47:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:53.409+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:54.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:54.151+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:54.373 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:53.958+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:54.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:54.371+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:55.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:55.159+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:54.919+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:55.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:55.358+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:56.159 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:55.880+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:56.323+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:56.158+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:56.865+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:57.293+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:57.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:47:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:47:57.282+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18877 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:47:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:57.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:58.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:57.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:58.303+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:58.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:58.238+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:59.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:58.805+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:59.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:47:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:47:59.344+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:47:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:47:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:47:59.272+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:47:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:47:59.792+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:00.606 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:00.347+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:00.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:00.282+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:00.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:01.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:01.373+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:01.311+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:02.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:01.735+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:02.330+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:02.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:02.284+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 18970 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:02.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:02.279+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:02.706+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:03.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:03.308+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:03.599 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:03.328+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:03.679+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:04.295+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:04.319+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:05.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:04.699+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:05.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:05.335+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:05.671 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:05.272+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:05.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:05.668+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:06.646 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:06.346+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:06.647 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:06.256+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:06.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:06.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:07.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:07.286+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19063 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:07.276+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:07.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:07.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:07.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:07.380+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:08.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:07.686+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:08.643 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:08.267+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:08.643 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:08.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:08.643 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:08.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:08.643 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:08.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:08.644 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:08.390+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:08.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:08.641+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:09.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:09.304+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:09.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:09.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:09.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:09.356+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:09.673+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:10.652 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:10.318+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:10.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:10.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:10.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:10.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:10.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:10.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:10.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:10.353+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:10.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:10.650+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:11.639 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:11.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:11.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:11.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:11.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:11.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:11.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:11.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:11.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:11.309+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:11.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:11.637+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:12.638 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:12.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:12.638 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:12.288+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19155 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:12.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:12.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:12.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:12.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:12.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:12.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:12.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:12.331+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:12.895 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:12.636+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:13.596 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:13.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:13.596 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:13.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:13.596 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:13.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:13.596 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:13.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:13.596 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:13.310+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:13.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:13.594+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:14.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:14.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:14.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:14.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:14.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:14.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:14.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:14.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:14.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:14.262+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:14.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:14.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:14.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:14.623+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:14.624+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:15.628 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:15.239+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:15.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:15.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:15.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:15.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:15.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:15.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:15.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:15.300+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:15.627+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:15.627+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:15.627+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:15.627+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:16.580 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:16.244+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:16.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:16.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:16.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:16.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:16.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:16.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:16.581 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:16.316+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:16.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:16.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:16.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:16.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:17.588 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:17.291+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19250 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:17.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:17.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:17.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:17.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:17.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:17.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:17.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:17.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:17.589 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:17.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:17.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:17.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:17.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:17.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:17.590 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:17.290+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:17.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:17.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:17.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:17.588+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:18.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:18.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:18.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:18.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:18.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:18.570 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:18.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:18.571 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:18.269+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:18.571 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:18.270+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:18.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:18.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:18.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:18.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:19.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:19.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:19.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:19.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:19.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:19.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:19.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:19.238+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:19.540 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:19.221+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:19.540 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:19.221+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:19.540 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:19.221+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:19.540 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:19.222+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:19.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:19.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:19.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:19.538+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:20.554 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:20.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:20.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:20.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:20.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:20.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:20.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:20.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:20.555 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:20.555 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:20.556 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:20.556 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:20.269+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:20.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:20.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:20.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:20.553+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:21.570 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:21.315+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:21.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:21.316+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:21.316+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:21.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:21.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:21.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:21.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:21.572 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:21.284+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:21.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:21.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:21.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:21.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:21.569+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:22.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:22.292+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19342 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:22.296+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:22.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:22.418 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:22.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:22.419 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:22.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:22.419 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:22.294+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:22.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:22.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:22.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:22.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:22.564+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:23.556 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:23.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:23.556 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:23.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:23.556 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:23.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:23.556 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:23.252+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:23.557 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:23.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:23.557 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:23.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:23.557 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:23.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:23.557 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:23.328+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:23.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:23.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:23.554+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:23.555+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:24.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:24.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:24.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:24.289+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:24.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:24.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:24.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:24.523+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:24.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:24.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:24.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:24.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:24.361+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:25.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:25.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:25.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:25.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:25.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:25.252+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:25.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:25.253+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:25.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:25.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:25.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:25.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:25.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:25.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:25.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:25.367+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:25.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:25.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:25.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:25.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:25.547+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:26.574 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:26.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:26.574 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:26.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:26.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:26.402+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:26.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:26.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:26.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:26.575 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:26.214+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:26.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:26.572+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:26.572+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:26.572+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:26.573+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:27.600 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:27.294+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19436 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:27.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:27.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:27.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:27.202+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:27.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:27.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:27.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:27.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:27.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:27.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:27.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:27.432+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:27.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:27.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:27.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:27.600+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:28.171+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:28.171+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:28.171+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:28.173+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:28.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:28.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:28.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:28.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:28.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:28.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:28.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:28.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:28.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:28.636+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:28.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:28.637+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:29.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:29.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:29.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:29.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:29.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:29.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:29.427+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:29.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:29.427+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:29.642+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:29.642+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:29.642+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:29.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:29.643+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:30.450 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:30.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:30.450 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:30.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:30.450 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:30.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:30.451 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:30.198+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:30.448+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:30.448+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:30.448+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:30.449+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:30.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:30.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:30.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:30.693+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:31.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:31.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:31.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:31.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:31.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:31.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:31.647 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:31.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:31.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:31.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:31.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:31.648 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:31.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:31.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:31.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:31.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:31.646+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:32.602 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:32.296+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19531 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:32.603 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:32.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:32.603 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:32.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:32.603 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:32.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:32.603 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:32.254+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:32.604 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:32.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:32.604 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:32.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:32.604 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:32.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:32.604 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:32.467+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:32.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:32.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:32.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:32.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:32.601+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:33.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:33.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:33.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:33.242+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:33.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:33.602 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:33.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:33.603 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:33.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:33.603 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:33.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:33.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:33.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:33.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:33.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:33.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:34.532 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:34.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:34.532 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:34.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:34.532 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:34.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:34.532 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:34.268+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:34.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:34.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:34.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:34.531+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:34.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:34.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:34.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:34.657+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:35.530 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:35.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:35.531 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:35.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:35.531 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:35.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:35.531 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:35.219+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:35.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:35.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:35.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:35.630+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:35.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:35.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:35.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:35.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:35.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:35.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:35.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:35.529+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:36.514 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:36.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:36.514 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:36.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:36.515 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:36.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:36.515 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:36.208+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:36.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:36.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:36.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:36.513+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:36.637+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:36.637+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:36.637+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:36.637+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:37.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:37.298+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19629 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:37.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:37.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:37.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:37.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:37.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:37.175+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:37.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:37.176+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:37.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:37.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:37.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:37.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:37.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:37.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:37.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:37.678+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:38.503 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:38.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:38.503 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:38.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:38.503 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:38.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:38.503 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:38.180+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:38.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:38.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:38.501+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:38.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:38.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:38.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:38.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:38.639+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:39.497 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:39.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:39.497 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:39.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:39.498 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:39.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:39.498 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:39.202+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:39.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:39.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:39.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:39.657+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:39.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:39.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:39.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:39.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:39.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:39.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:39.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:39.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:40.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:40.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:40.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:40.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:40.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:40.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:40.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:40.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:40.614+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:40.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:40.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:40.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:40.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:41.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:41.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:41.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:41.159+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:41.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:41.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:41.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:41.541+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:41.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:41.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:41.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:41.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:42.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:42.300+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19724 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:42.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:42.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:42.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:42.116+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:42.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:42.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:42.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:42.603+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:42.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:42.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:42.559+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:42.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:43.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:43.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:43.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:43.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:43.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:43.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:43.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:43.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:43.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:43.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:43.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:43.614+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:44.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:44.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:44.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:44.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:44.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:44.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:44.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:44.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:44.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:44.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:44.576+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:44.577+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:44.591+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:44.591+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:44.591+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:44.592+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:45.542 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:45.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:45.543 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:45.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:45.543 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:45.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:45.543 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:45.211+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:45.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:45.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:45.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:45.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:45.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:45.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:45.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:45.541+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:46.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:46.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:46.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:46.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:46.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:46.198+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:46.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:46.199+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:46.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:46.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:46.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:46.575+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:46.574+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:46.574+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:46.574+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:46.575+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:47.425 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:47.303+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19820 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:47.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:47.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:47.167+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:47.168+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:47.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:47.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:47.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:47.542+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:47.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:47.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:47.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:47.547+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:48.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:48.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:48.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:48.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:48.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:48.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:48.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:48.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:48.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:48.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:48.534+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:48.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:48.535+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:49.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:49.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:49.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:49.099+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:49.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:49.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:49.541+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:49.541+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:49.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:49.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:49.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:49.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:49.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:49.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:49.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:49.539+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:50.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:50.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:50.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:50.086+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:50.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:50.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:50.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:50.583+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:50.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:50.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:50.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:50.543+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:51.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:51.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:51.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:51.533+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:51.555+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:51.555+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:51.555+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:51.556+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:52.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:52.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:52.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.996+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:52.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:51.997+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:52.573 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:52.305+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 19915 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:52.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:52.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:52.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:52.831 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:52.580+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:52.832 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:52.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:52.832 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:52.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:52.832 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:52.571+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:52.832 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:52.572+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:52.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:52.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:53.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:52.991+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:53.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:52.992+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:53.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:53.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:53.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:53.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:53.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:53.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:53.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:53.626+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:53.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:53.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:53.977+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:53.977+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:54.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:54.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:54.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:54.623+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:54.515+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:54.515+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:54.515+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:54.515+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:55.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:55.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:55.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:55.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:55.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:55.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:55.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:55.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:55.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:55.607+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:55.902 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:55.608+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:55.903 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:55.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:55.903 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:55.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:55.903 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:55.532+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:55.903 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:55.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:56.019+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:56.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:56.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:56.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:56.606+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:56.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:56.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:56.550+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:56.551+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:57.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:57.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:57.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:57.309 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:57.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:57.656 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:48:57.307+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20013 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:48:57.657 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:57.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:57.657 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:57.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:57.657 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:57.529+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:57.657 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:57.530+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:57.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:57.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:57.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:57.654+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:57.655+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:58.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:58.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:58.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:58.004+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:58.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:58.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:58.698+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:58.698+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:58.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:58.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:58.537+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:58.538+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:48:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.007+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:59.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:59.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:59.663+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:48:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:48:59.663+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:48:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:59.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:59.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:59.549+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:48:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:48:59.549+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:48:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:48:59.981+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:00.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:00.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:00.645+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:00.646+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:00.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:00.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:00.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:00.547+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:01.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:00.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:00.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:00.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:00.955+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:01.623+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:01.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:01.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:01.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:01.558+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:02.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:01.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:02.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:01.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:02.311 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:01.943+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:02.312 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:01.944+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:02.568 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:02.310+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20108 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:02.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:02.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:02.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:02.590+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:02.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:02.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:02.567+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:02.568+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:02.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:02.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:02.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:02.975+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:03.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:03.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:03.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:03.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:03.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:03.572+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:03.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:03.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:03.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:03.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:03.617+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:03.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:03.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:03.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:03.982+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:04.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:04.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:04.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:04.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:04.532+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:04.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:04.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:04.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:04.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:05.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:05.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:05.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:05.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:05.013+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:05.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:05.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:05.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:05.499+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:05.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:05.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:05.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:05.553+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:06.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:06.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:06.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:06.020+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:06.539+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:06.539+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:06.539+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:06.539+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:06.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:06.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:06.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:06.506+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:07.313 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:07.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:07.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.034+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:07.314 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.035+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:07.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:07.311+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20204 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:07.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:07.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:07.538+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:07.539+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:07.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:07.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:07.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:07.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.997+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:07.997+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:08.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:08.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:08.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:08.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:08.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:08.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:08.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:08.520+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:08.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:08.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:08.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:08.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:08.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:08.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:08.464+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:09.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:08.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:08.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:08.992+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:08.993+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:09.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:09.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:09.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:09.512+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:09.559+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:09.559+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:09.559+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:09.560+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:10.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:10.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:10.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:10.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:10.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:10.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:10.511+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:10.512+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:10.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:10.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:10.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:10.606+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:11.262 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:11.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:11.262 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:11.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:11.262 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:11.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:11.262 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:11.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:11.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:11.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:11.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:11.568+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:11.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:11.569+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:11.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:11.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:11.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:11.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:11.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:11.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:11.517+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:12.315 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:12.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:12.316 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:12.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:12.316 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:12.023+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:12.316 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:12.024+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:12.617 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:12.313+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20298 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:12.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:12.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:12.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:12.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:12.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:12.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:12.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:12.563+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:12.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:12.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:12.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:12.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:12.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:13.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:13.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:13.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:13.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:13.038+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:13.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:13.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:13.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:13.594+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:13.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:13.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:13.662+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:13.663+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:14.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:14.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:14.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:14.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:14.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:14.008+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:14.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:14.009+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:14.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:14.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:14.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:14.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:15.044 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:14.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.045 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:14.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.045 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:14.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.045 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:14.700+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.043+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:15.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:15.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:15.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:15.613+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:15.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:15.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:15.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:15.651+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.995+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:15.996+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:16.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:16.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:16.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:16.673+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:16.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:16.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:16.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:16.596+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:17.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:16.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:17.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:16.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:17.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:16.983+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:17.318 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:16.983+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:17.631 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:17.316+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20395 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:17.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:17.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:17.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:17.630+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:17.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:17.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:17.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:17.646+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:18.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:17.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:17.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:17.973+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:17.974+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:18.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:18.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:18.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:18.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:18.666+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:18.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:18.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:18.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:18.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:19.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:18.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:18.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:18.940+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:18.941+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:19.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:19.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:19.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:19.662+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:19.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:19.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:19.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:19.718+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:19.939+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:19.940+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:20.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:20.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:20.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:20.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:20.757 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:20.678+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:20.758 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:20.679+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:20.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:20.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:20.941+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:20.942+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:20.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:20.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:20.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:20.756+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:21.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:21.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:21.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:21.634+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:21.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:21.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:21.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:21.804+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:22.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:21.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:22.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:21.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:22.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:21.927+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:22.320 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:21.928+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:22.588 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:22.318+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20489 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:22.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:22.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:22.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:22.587+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:22.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:22.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:22.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:22.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:22.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:22.815+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:22.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:22.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:22.955+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:22.956+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:23.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:23.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:23.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:23.586+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:23.795+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:24.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:23.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:23.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:23.946+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:23.946+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:24.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:24.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:24.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:24.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:24.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:24.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:24.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:24.586+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:24.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:24.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:24.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:24.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:24.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:24.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:24.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:24.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:24.928+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:24.929+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:25.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:25.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:25.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:25.617+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:25.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:25.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:25.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:25.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:25.752+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:25.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:25.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:25.978+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:25.979+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:26.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:26.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:26.608+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:26.609+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:26.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:26.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:26.722+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:26.722+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:27.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:26.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:27.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:26.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:27.322 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:26.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:27.323 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:26.952+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:27.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:27.320+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20585 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:27.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:27.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:27.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:27.568+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:27.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:27.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:27.726+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:27.727+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:27.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:27.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:27.974+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:27.975+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:28.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:28.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:28.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:28.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:28.703+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:28.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:28.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:28.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:28.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:28.929 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:28.552+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:29.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:29.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:29.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:29.001+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:29.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:29.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:29.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:29.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:29.710 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:29.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:29.710 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:29.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:30.009 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:29.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:30.009 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:29.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:30.009 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:29.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:30.010 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:29.702+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:30.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:30.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:30.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:30.007+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:30.008+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:30.735 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:30.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:30.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:30.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:30.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:30.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:30.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:30.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:31.053 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:30.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:31.053 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:30.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:31.054 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:30.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:31.054 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:30.734+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:31.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:31.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:31.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:31.052+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:31.711 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:31.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:31.711 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:31.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:31.711 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:31.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:31.712 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:31.445+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:32.095 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:31.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:32.095 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:31.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:32.095 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:31.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:32.096 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:31.750+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:32.350 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:32.322+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20680 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:32.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:32.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:32.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:32.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:32.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:32.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:32.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:32.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:32.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:32.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:33.075 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:32.707+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:33.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:32.707+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:33.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:32.707+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:33.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:32.708+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:33.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:33.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:33.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:33.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:33.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:33.708 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:33.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:33.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:33.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:33.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:33.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:33.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:33.439+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:34.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:33.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.072 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:33.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.072 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:33.706+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.072 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:33.708+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:34.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:34.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:34.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:34.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:34.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:34.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:34.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:34.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:34.693+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:34.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:34.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:34.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:34.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:34.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:34.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.036+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:35.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:35.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:35.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:35.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:35.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:35.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:35.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:35.444+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:36.002 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:35.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.002 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:35.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.002 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:35.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:35.703+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:35.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:36.000+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:36.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:36.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:36.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:36.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:36.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:36.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:36.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:36.654+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:37.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:37.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:37.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:37.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:37.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:37.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:37.327 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:37.050+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:37.626 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:37.325+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20778 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:37.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:37.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:37.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:37.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:37.627 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:37.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:37.627 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:37.437+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:37.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:37.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:37.624+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:37.625+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:38.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:38.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:38.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:38.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:38.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:38.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:38.471+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:38.472+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:38.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:38.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:38.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:38.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:38.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:39.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:39.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:39.051+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:39.052+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:39.713 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:39.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:39.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:39.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:39.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:39.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:39.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:39.454+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:40.083 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:39.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:40.084 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:39.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:40.084 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:39.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:40.084 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:39.712+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:40.419 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:40.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:40.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:40.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:40.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:40.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:40.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:40.082+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:40.415+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:40.415+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:40.415+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:40.416+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:41.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:40.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:40.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:40.747+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:40.747+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:41.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:41.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:41.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:41.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:41.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:41.712 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:41.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:41.712 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:41.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:41.712 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:41.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:41.712 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:41.444+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:41.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:41.742+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:41.742+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:41.742+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:41.982 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:41.743+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:42.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:42.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:42.329 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:42.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:42.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:42.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:42.330 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:42.076+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:42.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:42.327+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20873 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:42.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:42.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:42.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:42.474+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:43.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:42.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:42.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:42.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:42.714+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:43.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:43.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:43.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:43.123+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:43.681+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:43.681+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:43.681+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:43.681+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:43.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:43.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:43.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:43.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:43.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:43.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:43.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:43.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:44.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:44.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:44.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:44.106+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:44.691+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:44.691+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:44.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:44.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:44.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:44.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:44.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:44.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:45.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:45.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:45.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:45.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:45.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:45.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:45.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:45.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:45.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:45.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:45.714 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:45.460+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:46.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:45.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:45.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:45.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:45.713+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:46.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:46.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:46.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:46.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:46.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:46.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:46.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:46.725+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:46.725+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:46.725+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:46.726+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:47.456 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:47.331+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 20966 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:47.456 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:47.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:47.457 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:47.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:47.457 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:47.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:47.457 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:47.187+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:47.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:47.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:47.767 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:47.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:47.767 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:47.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:47.767 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:47.485+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:47.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:47.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:47.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:47.766+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:48.534 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:48.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:48.534 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:48.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:48.534 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:48.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:48.535 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:48.194+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:48.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:48.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:48.719+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:48.720+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:48.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:48.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:48.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:48.534+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:49.580 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:49.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:49.580 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:49.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:49.580 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:49.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:49.580 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:49.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:49.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:49.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:49.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:49.689+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:49.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:49.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:49.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:49.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:49.579+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:50.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:50.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:50.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:50.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:50.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:50.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:50.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:50.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:50.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:50.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:50.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:50.732+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:50.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:50.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:50.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:50.589+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:51.606 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:51.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:51.606 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:51.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:51.606 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:51.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:51.606 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:51.207+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:51.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:51.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:51.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:51.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:51.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:51.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:51.605+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:51.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:51.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:51.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:51.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:51.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:51.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:51.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:51.696+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:52.459 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:52.332+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21057 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:52.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:52.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:52.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:52.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:52.460 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:52.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:52.460 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:52.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:52.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:52.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:52.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:52.623+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:52.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:52.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:52.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:52.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:53.607 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:53.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:53.607 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:53.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:53.607 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:53.213+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:53.607 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:53.214+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:53.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:53.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:53.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:53.651+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:53.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:53.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:53.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:53.661+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:54.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:54.223+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:54.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:54.223+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:54.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:54.223+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:54.605 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:54.225+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:54.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:54.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:54.659+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:54.660+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:54.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:54.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:55.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:55.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:55.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:55.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:55.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:55.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:55.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:55.219+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:55.642+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:55.643+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:55.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:55.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:55.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:55.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:55.564+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:56.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:56.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:56.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:56.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:56.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:56.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:56.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:56.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:56.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:56.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:56.574+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:56.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:56.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:56.621+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:56.621+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:57.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:49:57.334+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21147 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:49:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:57.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:57.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:57.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:57.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:57.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:57.650+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:57.650+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:57.650+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:57.651+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:57.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:57.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:57.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:57.545+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:58.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:58.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:58.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:58.145+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:58.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:58.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:58.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:58.666+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:58.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:58.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:58.523+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:58.524+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:59.485 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:59.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:59.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:59.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:59.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:59.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:49:59.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:49:59.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:59.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:59.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:59.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:59.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:49:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:49:59.484+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:49:59.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:59.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:59.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:59.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:59.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:59.637+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:49:59.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:49:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:49:59.638+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:00.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:00.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:00.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:00.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:00.477 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:00.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:00.477 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:00.202+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:00.474+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:00.474+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:00.474+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:00.475+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:00.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:00.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:00.627+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:00.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:00.628+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:01.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:01.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:01.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:01.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:01.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:01.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:01.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:01.216+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:01.622+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:01.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:01.622+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:01.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:01.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:01.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:01.481+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:02.461 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:02.337+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21232 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:02.462 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:02.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:02.462 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:02.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:02.462 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:02.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:02.462 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:02.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:02.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:02.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:02.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:02.620+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:02.478+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:02.479+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:02.479+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:02.480+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:03.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:03.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:03.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:03.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:03.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:03.193+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:03.486 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:03.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:03.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:03.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:03.653+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:03.654+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:03.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:03.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:03.525+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:03.526+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:04.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:04.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:04.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:04.669+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:04.670+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:04.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:04.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:04.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:04.563+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:04.564+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:05.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:05.121+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:05.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:05.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:05.658+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:05.659+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:05.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:05.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:05.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:05.598+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:06.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:06.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:06.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:06.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:06.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:06.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:06.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:06.553+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:06.554+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:06.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:06.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:06.629+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:06.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:06.630+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:07.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:07.339+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21320 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:07.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:07.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:07.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:07.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:07.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:07.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:07.364 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:07.098+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:07.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:07.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:07.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:07.600+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:07.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:07.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:07.573+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:07.573+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:08.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:08.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:08.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:08.124+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:08.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:08.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:08.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:08.575+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:08.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:08.576+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:08.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:08.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:08.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:08.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:08.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:08.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:08.598+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:09.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:09.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:09.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:09.135+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:09.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:09.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:09.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:09.529+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:09.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:09.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:09.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:10.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:10.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:10.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:10.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:10.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:10.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:10.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:10.561+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:10.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:10.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:10.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:10.596+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:11.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:11.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:11.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:11.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:11.543+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:11.544+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:11.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:11.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:11.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:11.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:11.562+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:11.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:11.563+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:12.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:12.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:12.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:12.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:12.354 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:12.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:12.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:12.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:12.355 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:12.341+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21408 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:12.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:12.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:12.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:12.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:12.577+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:12.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:12.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:12.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:12.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:13.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:13.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:13.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:13.129+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:13.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:13.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:13.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:13.538+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:13.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:13.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:13.556+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:13.557+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:14.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:14.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:14.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:14.082+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:14.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:14.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:14.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:14.562+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:14.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:14.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:14.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:14.593+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:15.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:15.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:15.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:15.075+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:15.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:15.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:15.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:15.528+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:15.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:15.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:15.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:15.584+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:16.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:16.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:16.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:16.103+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:16.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:16.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:16.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:16.480+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:16.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:16.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:16.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:16.584+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:17.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:17.344+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21507 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:17.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:17.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:17.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:17.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:17.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:17.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:17.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:17.578+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:17.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:17.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:17.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:17.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:17.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:17.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:17.442+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:18.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:18.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:18.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:18.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:18.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:18.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:18.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:18.457+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:18.607+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:18.607+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:18.607+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:18.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:18.608+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:19.495 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:19.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:19.495 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:19.183+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:19.495 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:19.183+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:19.496 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:19.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:19.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:19.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:19.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:19.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:19.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:19.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:19.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:19.626+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:20.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:20.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:20.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:20.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:20.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:20.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:20.473 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:20.228+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:20.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:20.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:20.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:20.471+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:20.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:20.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:20.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:20.652+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:21.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:21.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:21.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:21.227+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:21.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:21.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:21.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:21.432+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:21.686+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:21.686+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:21.686+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:21.687+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:22.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:22.346+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21601 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:22.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:22.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:22.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:22.456+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:22.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:22.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:22.242+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:22.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:22.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:22.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:22.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:22.736+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:23.464 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:23.464 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:23.464 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:23.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:23.464 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:23.462+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:23.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:23.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:23.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:23.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:23.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:23.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:23.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:23.212+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:24.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:23.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:23.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:23.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:23.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:24.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:24.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:24.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:24.174+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:24.735 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:24.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:24.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:24.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:24.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:24.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:24.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:24.454+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:24.990 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:24.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.990 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:24.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.990 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:24.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:24.990 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:24.734+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:25.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:25.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:25.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:25.152+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:25.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:25.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:25.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:25.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:25.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:25.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:25.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:25.429+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:26.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:25.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:26.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:25.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:26.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:25.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:26.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:25.736+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:26.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:26.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:26.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:26.155+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:26.776 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:26.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:26.777 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:26.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:26.777 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:26.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:26.777 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:26.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:27.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:26.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:27.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:26.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:27.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:26.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:27.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:26.775+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:27.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:27.348+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21698 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:27.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:27.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:27.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:27.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:27.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:27.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:27.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:27.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:27.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:27.443+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:27.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:27.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:28.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:27.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:28.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:27.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:28.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:27.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:28.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:27.799+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:28.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:28.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:28.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:28.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:28.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:28.696 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:28.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:28.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:28.441+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:28.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:28.441+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:28.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:28.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:28.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:28.826+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:29.466 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:29.177+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:29.466 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:29.177+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:29.466 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:29.177+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:29.466 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:29.178+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:29.782 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:29.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:29.782 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:29.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:29.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:29.463+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:29.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:29.464+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:30.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:29.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:30.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:29.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:30.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:29.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:30.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:29.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:30.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:30.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:30.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:30.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:30.782 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:30.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:30.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:30.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:30.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:30.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:30.783 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:30.495+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:31.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:30.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:30.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:30.780+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.134 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:30.781+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:31.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:31.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:31.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:31.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:31.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:31.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:31.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:31.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:31.534+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:31.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:31.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:31.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:31.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:31.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:31.736+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:32.476 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:32.352+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21792 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:32.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:32.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:32.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:32.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:32.476 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:32.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:32.477 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:32.177+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:32.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:32.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:32.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:32.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:32.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:32.549+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:32.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:32.550+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:33.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:32.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:32.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:32.737+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:32.737+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:33.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:33.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:33.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:33.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:33.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:33.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:33.713+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:33.714+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:33.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:33.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:33.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:33.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:33.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:33.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:33.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:34.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:34.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:34.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:34.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:34.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:34.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:34.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:34.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:34.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:34.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:34.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:34.716+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:35.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:35.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:35.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:35.088+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:35.761 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:35.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:35.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:35.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:35.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:35.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:35.762 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:35.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:36.043 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:35.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:36.044 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:35.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:36.044 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:35.759+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:36.044 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:35.760+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:36.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:36.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:36.041+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:36.043+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:36.740 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:36.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:36.741 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:36.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:36.741 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:36.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:36.741 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:36.483+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:37.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:36.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:37.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:36.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:37.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:36.738+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:37.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:36.739+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:37.355 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:37.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:37.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:37.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:37.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:37.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:37.356 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:37.006+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:37.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:37.353+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21891 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:37.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:37.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:37.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:37.453+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:38.012 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:37.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:38.013 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:37.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:38.013 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:37.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:38.013 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:37.705+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:38.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:38.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:38.011+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:38.012+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:38.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:38.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:38.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:38.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:38.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:38.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:38.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:38.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:39.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:38.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:39.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:38.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:39.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:38.751+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:39.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:38.752+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:39.410 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:39.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:39.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:39.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:39.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:39.014+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:39.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:39.015+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:39.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:39.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:39.408+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:39.410+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:40.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:39.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:40.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:39.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:40.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:39.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:40.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:39.708+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:40.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:40.350 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:40.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:40.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:40.351 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:40.001+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:40.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:40.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:40.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:40.416+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:40.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:40.417+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:41.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:40.739+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:41.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:40.739+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:41.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:40.739+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:41.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:40.741+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:41.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:41.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:41.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:41.029+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:41.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:41.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:41.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:41.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:41.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:41.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:41.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:41.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:42.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:41.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:42.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:41.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:42.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:41.724+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:42.076 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:41.725+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:42.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:42.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:42.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:42.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:42.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:42.074+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:42.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:42.075+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:42.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:42.356+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 21985 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:42.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:42.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:42.408+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:43.089 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:42.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:43.089 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:42.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:43.089 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:42.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:43.090 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:42.747+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:43.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:43.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:43.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:43.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:43.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:43.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:43.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:43.088+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:43.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:43.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:43.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:43.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:43.378+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:44.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:43.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:44.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:43.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:44.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:43.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:44.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:43.700+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:44.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:44.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:44.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:44.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:44.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:44.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:44.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:44.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:44.332+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:45.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:44.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:45.161 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:44.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:45.161 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:44.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:45.161 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:44.734+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:45.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:45.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:45.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:45.333+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:45.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:45.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:45.158+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:45.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:45.159+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:46.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:45.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.153 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:45.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.153 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:45.685+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.153 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:45.685+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:46.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:46.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:46.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:46.151+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:46.345+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:46.345+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:46.345+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:46.346+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:46.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:46.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:46.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:46.639+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:46.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:46.640+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:47.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:47.358+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22082 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:47.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:47.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:47.326+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:47.327+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:47.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:47.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:47.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:47.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:47.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:47.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:47.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:47.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:48.628 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:48.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:48.628 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:48.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:48.628 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:48.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:48.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:48.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:48.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:48.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:48.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:48.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:48.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:48.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:48.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:48.368+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:48.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:48.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:48.626+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:48.627+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:49.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:49.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:49.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:49.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:49.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:49.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:49.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:49.408+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:49.620 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:49.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:49.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:49.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:49.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:49.206+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:49.621 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:49.207+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:49.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:49.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:50.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:50.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:50.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:50.174+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:50.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:50.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:50.391+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:50.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:50.391+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:50.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:50.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:50.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:50.586+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:51.402 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:51.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:51.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:51.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:51.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:51.142+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:51.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:51.143+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:51.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:51.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:51.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:51.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:51.612+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:51.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:51.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:51.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:51.400+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:51.401+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:52.426 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:52.361+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22176 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:52.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:52.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:52.149+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:52.150+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:52.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:52.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:52.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:52.614+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:52.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:52.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:52.444+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:52.445+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:53.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:53.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:53.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:53.147+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:53.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:53.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:53.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:53.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:53.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:53.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:53.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:53.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:53.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:53.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:53.657+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:53.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:53.658+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:54.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:54.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:54.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:54.118+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:54.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:54.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:54.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:54.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:54.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:54.609+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:54.609+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:54.609+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:54.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:54.610+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:55.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:55.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:55.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:55.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:55.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:55.077+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:55.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:55.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:55.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:55.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:55.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:55.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:55.619+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:55.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:55.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:55.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:55.403+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:56.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:56.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:56.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:56.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:56.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:56.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:56.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:56.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:56.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:56.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:56.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:56.399+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:56.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:56.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:56.586+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:56.587+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:57.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:57.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:57.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:57.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:57.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:57.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:57.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:50:57.363+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22274 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:50:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:57.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:57.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:57.385+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:57.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:57.386+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:57.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:57.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:57.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:57.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:57.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:57.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:57.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:57.592+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:58.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:58.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:58.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:58.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:58.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:58.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:58.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:58.007+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:58.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:58.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:58.406+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:58.406+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:58.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:58.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:58.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:58.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:58.600+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:59.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:59.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:59.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:59.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:59.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:59.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:50:59.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:50:59.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:59.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:59.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:59.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:50:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:50:59.405+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:50:59.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:59.612+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:59.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:59.612+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:59.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:59.612+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:50:59.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:50:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:50:59.613+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:00.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:00.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:00.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:00.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:00.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:01.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:01.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:01.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:01.423 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:00.973+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:01.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:01.421+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:01.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:01.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:01.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:01.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:01.655+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:02.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:02.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:02.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:02.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:02.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:02.015+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:02.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:02.016+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:02.635 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:02.366+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22368 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:02.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:02.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:02.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:02.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:02.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:02.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:02.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:02.436+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:02.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:02.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:02.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:02.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:02.634+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:03.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:03.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:03.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:03.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:03.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:03.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:03.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:03.043+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:03.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:03.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:03.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:03.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:03.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:03.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:03.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:03.397+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:03.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:03.644+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:03.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:03.644+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:03.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:03.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:03.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:03.645+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:04.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:04.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:04.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:04.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:04.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:04.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:04.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:04.071+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:04.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:04.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:04.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:04.384+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:04.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:04.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:04.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:04.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:04.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:04.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:04.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:05.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:05.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:05.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:05.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:05.101+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:05.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:05.411+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:05.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:06.054 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:05.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:05.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:05.678+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.055 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:05.679+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:06.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:06.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:06.052+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:06.053+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:06.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:06.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:06.431+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:06.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:06.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:06.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:06.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:06.677+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:07.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:07.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:07.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:07.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:07.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:07.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:07.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:07.053+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:07.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:07.368+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22465 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:07.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:07.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:07.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:07.420+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:08.022 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:07.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:08.022 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:07.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:08.023 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:07.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:08.023 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:07.702+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:08.378 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:08.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:08.378 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:08.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:08.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:08.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:08.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:08.021+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:08.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:08.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:08.376+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:08.377+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:09.035 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:08.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:09.036 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:08.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:09.036 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:08.679+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:09.036 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:08.679+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:09.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:09.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:09.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:09.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:09.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:09.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:09.345 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:09.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:09.343+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:09.344+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:10.005 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:09.721+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:10.006 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:09.721+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:10.006 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:09.721+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:10.006 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:09.722+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:10.306 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:10.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:10.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.003+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:10.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.004+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:10.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:10.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:10.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:10.305+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:11.070 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:11.070 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:11.070 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.958+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:11.071 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:10.958+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:11.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:10.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:11.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:10.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:11.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:10.717+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:11.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:10.718+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:11.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:11.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:11.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:11.302+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:11.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:11.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:11.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:12.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:11.963+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:12.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:11.746+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:12.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:12.371+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22559 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:12.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:12.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:12.335+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:12.336+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:12.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:12.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:12.919+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:12.920+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:12.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:12.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:12.732+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:12.733+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:13.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:13.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:13.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:13.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:13.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:13.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:13.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:13.360+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:13.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:13.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:13.710+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:13.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:13.710+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:14.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:13.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:14.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:13.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:14.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:13.926+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:14.368 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:13.927+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:14.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:14.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:14.366+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:14.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:14.366+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:14.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:14.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:14.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:15.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:14.923+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:15.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:14.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:15.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:14.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:15.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:14.693+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:15.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:14.694+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:15.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:15.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:15.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:15.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:15.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:15.918+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:15.918+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:15.918+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:15.918+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:15.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:15.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:15.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:15.713+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:16.732 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:16.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:16.733 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:16.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:16.733 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:16.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:16.733 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:16.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:16.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:16.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:16.962+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:17.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:16.963+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:17.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:16.731+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:17.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:16.731+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:17.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:16.731+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:17.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:16.732+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:17.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:17.375+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22658 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:17.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:17.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:17.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:17.493+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:17.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:17.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:17.922+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:17.923+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:17.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:17.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:17.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:17.737+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:18.765 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:18.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:18.766 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:18.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:18.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:18.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:18.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:18.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:18.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:18.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:18.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:18.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:18.764+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:18.765+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:18.880+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:18.880+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:18.880+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:19.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:18.881+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:19.875 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:19.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:19.875 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:19.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:19.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:19.734+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:19.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:19.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:19.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:19.876 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:19.525+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:19.871+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:19.871+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:19.871+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:19.871+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:20.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:20.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:20.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:20.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:20.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:20.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:20.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:20.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:20.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:20.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:20.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:20.750+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:20.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:20.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:20.886+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:20.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:21.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:21.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:21.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:21.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:21.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:21.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:21.903 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:21.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:21.903 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:21.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:21.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:21.715+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:21.904 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:21.716+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:21.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:22.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:21.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:21.900+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:21.902+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:22.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:22.376+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22752 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:22.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:22.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:22.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:22.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:22.761+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:22.762+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:22.762+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:22.762+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:23.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:22.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:22.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:22.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:22.885+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:23.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:23.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:23.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:23.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:23.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:23.460+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:23.612 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:23.461+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:23.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:23.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:23.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:23.857+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:23.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:23.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:23.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:23.811+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:24.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:24.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:24.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:24.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:24.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:24.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:24.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:24.499+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:24.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:24.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:24.899+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:24.900+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:25.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:24.859+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:25.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:24.859+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:25.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:24.859+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:25.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:24.859+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:25.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:25.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:25.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:25.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:25.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:25.516+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:25.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:25.516+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:25.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:25.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:25.902+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:25.903+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:25.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:25.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:25.890+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:25.891+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:26.846 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:26.847 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:26.847 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:26.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:26.847 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:26.554+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:26.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:26.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:26.844+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:26.845+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:26.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:26.875+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:27.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:27.379+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22849 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:27.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:27.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:27.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:27.586+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:27.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:27.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:27.870+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:27.870+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:27.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:27.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:27.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:27.835+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:28.844 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:28.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:28.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:28.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:28.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:28.554+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:28.845 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:28.555+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:28.861+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:28.861+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:28.861+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:28.861+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:28.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:28.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:28.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:28.843+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:29.853 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:29.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.853 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:29.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:29.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:29.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:29.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:29.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:29.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:29.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:29.600+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:29.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:29.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:29.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:29.852+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:30.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:30.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:30.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:30.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:30.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:30.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:30.825 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:30.558+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:30.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:31.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:30.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:31.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:30.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:31.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:30.847+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:31.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:30.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:31.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:30.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:31.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:30.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:31.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:30.823+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:31.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:31.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:31.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:31.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:31.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:31.555+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:31.827 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:31.556+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:31.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:31.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:31.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:31.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:32.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:31.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:32.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:31.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:32.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:31.873+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:32.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:31.874+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:32.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:32.381+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 22943 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:32.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:32.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:32.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:32.527+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:32.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:32.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:32.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:32.857+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:33.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:32.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:33.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:32.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:33.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:32.856+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:33.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:32.857+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:33.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:33.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:33.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:33.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:33.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:33.533+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:33.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:33.534+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:33.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:33.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:33.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:33.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:33.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:34.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:33.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:34.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:33.902+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:34.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:33.903+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:34.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:34.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:34.877 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:34.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:34.538+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:34.538+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:34.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:34.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:34.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:34.876+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:34.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:34.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:34.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:34.888+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:35.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:35.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:35.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:35.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:35.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:35.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:35.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:35.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:35.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:35.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:36.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:35.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:36.179 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:35.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:36.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:35.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:36.179 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:35.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:36.180 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:35.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:36.180 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:35.858+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:36.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:36.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:36.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:36.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:36.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:36.577+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:36.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:36.578+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:36.881+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:36.881+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:36.881+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:36.881+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:36.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:36.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:36.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:36.852+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:37.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:37.384+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23042 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:37.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:37.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:37.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:37.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:37.584+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:37.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:37.584+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:37.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:37.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:37.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:37.867+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:37.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:37.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:37.900+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:37.901+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:38.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:38.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:38.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:38.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:38.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:38.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:38.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:38.582+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:38.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:38.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:38.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:38.863+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:38.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:38.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:38.892+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:39.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:38.893+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:39.847 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:39.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:39.848 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:39.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:39.848 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:39.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:39.848 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:39.578+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:39.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:39.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:39.884+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:39.885+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:40.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:39.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:40.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:39.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:40.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:39.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:40.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:39.846+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:40.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:40.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:40.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:40.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:40.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:40.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:40.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:40.606+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:40.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:40.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:40.887+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:41.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:40.887+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:41.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:40.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:41.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:40.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:41.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:40.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:41.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:40.879+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:41.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:41.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:41.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:41.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:41.861 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:41.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:41.862 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:41.586+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:42.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:41.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:42.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:41.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:42.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:41.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:42.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:41.919+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:42.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:41.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:42.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:41.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:42.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:41.859+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:42.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:41.860+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:42.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:42.386+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23136 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:42.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:42.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:42.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:42.570+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:42.873+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:42.873+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:42.873+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:42.874+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:42.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:42.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:42.898+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:43.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:42.899+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:43.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:43.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:43.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:43.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:43.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:43.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:43.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:43.573+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:43.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:43.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:43.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:44.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:43.897+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:44.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:43.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:44.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:43.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:44.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:43.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:44.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:43.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:44.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:44.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:44.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:44.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:44.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:44.542+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:44.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:44.543+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:44.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:44.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:44.920+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:44.920+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:44.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:44.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:44.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:44.852+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:45.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:45.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:45.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:45.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:45.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:45.572+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:45.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:45.572+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:45.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:45.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:45.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:45.835+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:46.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:45.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:45.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:45.882+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:46.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:45.883+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:46.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:46.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:46.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:46.590+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:46.591+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:46.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:46.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.850 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:46.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:46.851 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:46.831+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:47.175 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:46.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:47.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:46.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:47.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:46.847+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:47.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:46.848+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:47.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:47.389+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23232 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:47.548+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:47.549+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:47.885+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:47.885+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:47.885+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:48.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:47.886+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:48.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:47.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:48.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:47.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:48.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:47.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:48.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:47.872+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:48.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:48.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:48.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:48.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:48.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:48.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:48.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:48.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:48.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:48.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:48.895+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:48.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:49.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:48.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:49.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:48.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:49.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:48.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:49.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:48.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:49.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:49.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:49.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:49.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:49.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:49.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:49.851 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:49.495+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:49.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:49.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:49.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:49.854+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:49.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:49.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:49.849+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:49.849+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:50.816 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:50.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:50.817 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:50.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:50.817 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:50.528+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:50.817 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:50.529+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:51.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:50.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:51.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:50.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:51.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:50.814+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:51.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:50.815+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:51.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:50.840+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:51.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:50.840+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:51.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:50.840+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:51.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:50.840+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:51.837 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:51.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:51.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:51.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:51.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:51.486+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:51.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:51.487+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:51.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:51.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:51.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:52.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:51.836+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:52.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:51.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:52.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:51.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:52.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:51.837+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:52.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:51.838+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:52.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:52.392+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23326 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:52.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:52.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:52.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:52.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:52.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:52.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:52.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:52.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:52.793+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:52.794+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:53.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:52.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:52.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:52.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:52.886+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:53.865 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:53.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.866 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:53.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.866 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:53.748+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:53.866 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:53.750+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:53.866 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:53.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:53.866 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:53.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:53.867 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:53.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:53.867 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:53.528+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:54.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:53.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:53.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:53.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:53.864+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:54.775 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:54.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:54.776 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:54.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:54.776 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:54.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:54.776 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:54.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:54.773+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:54.773+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:54.773+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:54.774+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:54.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:54.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:54.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:54.836+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:55.747 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:55.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:55.747 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:55.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:55.747 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:55.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:55.748 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:55.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:55.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:55.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:55.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:55.867+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:55.745+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:55.745+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:55.745+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:55.746+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:56.730 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:56.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:56.730 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:56.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:56.730 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:56.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:56.730 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:56.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:56.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:56.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:56.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:56.729+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:56.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:56.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:56.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:56.898+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:57.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:51:57.394+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23424 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:51:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:57.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:57.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:57.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:57.468+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:57.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:57.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:57.764+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:57.764+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:57.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:57.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:57.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:57.906+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:58.751 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:58.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:58.751 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:58.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:58.751 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:58.494+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:58.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:58.495+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:58.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:58.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:58.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:51:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:58.750+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:58.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:58.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:58.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:51:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:58.906+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:51:59.742 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:59.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:59.742 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:59.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:59.742 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:59.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:51:59.742 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:51:59.489+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:59.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:59.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:59.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:51:59.923+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:59.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:59.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:59.740+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:51:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:51:59.741+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:00.894 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:00.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:00.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:00.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:00.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:00.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:00.895 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:00.502+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:00.895 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:00.705+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.895 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:00.705+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.896 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:00.705+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:00.896 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:00.706+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:00.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:00.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:00.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:00.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:01.722 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:01.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:01.722 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:01.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:01.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:01.658+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:01.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:01.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:01.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:01.723 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:01.463+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:02.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:01.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:02.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:01.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:02.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:01.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:02.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:01.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:02.654 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:02.398+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23518 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:02.654 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:02.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:02.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:02.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:02.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:02.452+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:02.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:02.453+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:02.919 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:02.653+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:02.919 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:02.653+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:02.920 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:02.653+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:02.920 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:02.654+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:02.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:02.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:02.918+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:02.919+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:03.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:03.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:03.614 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:03.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:03.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:03.435+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:03.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:03.436+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:03.694+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:03.694+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:03.694+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:03.695+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:03.947+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:03.947+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:03.947+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:03.948+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:04.716 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:04.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:04.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:04.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:04.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:04.433+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:04.717 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:04.434+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:04.979 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:04.714+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:04.979 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:04.714+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:04.979 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:04.714+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:04.979 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:04.715+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:04.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:04.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:05.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:04.977+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:04.977+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:05.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:05.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:05.752 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:05.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:05.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:05.479+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:05.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:05.480+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:06.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:05.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:06.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:05.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:06.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:05.750+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:06.012 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:05.751+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:06.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:06.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:06.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:06.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:06.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:06.010+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:06.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:06.011+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:06.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:06.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:06.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:06.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:07.014 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:06.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.014 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:06.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.015 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:06.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.015 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:06.727+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:07.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:07.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:07.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:07.402 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.013+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:07.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:07.400+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23614 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:07.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:07.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:07.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:07.525+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:07.984 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:07.725+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.984 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:07.725+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.984 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:07.725+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:07.984 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:07.725+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.982+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.982+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.982+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:07.983+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:08.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:08.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:08.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:08.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:08.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:08.742+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:08.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:08.742+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:08.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:08.742+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:08.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:08.743+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:09.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:09.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:09.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:09.007+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:09.739 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:09.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:09.739 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:09.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:09.739 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:09.478+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:09.739 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:09.479+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:10.053 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:09.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.053 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:09.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.053 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:09.737+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.053 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:09.737+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:10.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:10.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:10.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:10.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:10.051+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:10.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:10.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:10.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:10.509+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:10.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:10.699+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:10.699+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:10.699+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:10.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:10.699+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:11.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:11.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:11.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:11.027+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:11.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:11.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:11.728+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:11.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:11.729+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:11.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:11.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:11.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:11.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:12.404 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:12.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:12.405 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:12.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:12.405 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:12.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:12.405 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:12.040+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:12.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:12.402+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23707 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:12.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:12.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:12.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:12.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:13.060 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:12.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.060 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:12.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.060 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:12.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.060 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:12.689+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:13.058+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:13.058+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:13.058+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:13.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:13.058+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:13.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:13.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:13.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:13.532+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:13.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:13.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:13.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:13.720+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:13.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:13.720+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:14.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:14.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:14.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:14.098+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:14.743 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:14.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:14.744 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:14.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:14.744 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:14.485+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:14.744 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:14.486+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:15.080 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:14.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.081 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:14.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.081 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:14.741+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.081 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:14.742+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:15.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:15.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:15.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:15.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:15.079+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:15.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:15.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:15.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:15.752+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:15.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:15.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:15.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:15.525+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:16.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:16.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:16.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:16.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:16.052+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:16.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:16.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:16.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:16.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:16.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:16.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:16.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:16.545+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:17.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:16.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:17.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:16.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:17.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:16.797+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:17.102 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:16.799+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:17.407 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:17.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:17.408 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:17.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:17.408 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:17.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:17.408 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:17.100+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:17.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:17.405+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23806 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:17.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:17.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:17.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:17.524+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:17.524+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:18.136 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:17.845+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.137 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:17.845+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.137 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:17.845+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.137 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:17.846+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:18.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:18.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:18.134+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:18.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:18.135+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:18.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:18.814+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:18.814+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:18.814+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:18.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:18.814+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:18.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:18.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:18.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:18.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:19.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:19.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:19.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:19.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:19.105+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:19.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:19.771+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:19.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:19.771+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:19.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:19.771+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:19.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:19.771+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:19.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:19.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:19.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:19.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:19.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:19.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:19.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:19.601+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:20.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:20.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:20.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:20.059+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:20.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:20.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:20.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:20.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:20.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:20.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:20.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:20.747+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:20.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:20.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:20.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:20.572+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:21.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:21.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:21.057+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:21.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:21.058+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:21.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:21.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:21.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:21.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:21.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:21.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:21.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:21.537+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:22.104 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:21.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:22.104 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:21.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:22.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:21.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:22.105 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:21.795+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:22.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:22.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:22.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:22.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:22.410 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:22.102+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:22.411 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:22.103+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:22.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:22.408+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23900 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:22.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:22.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:22.491+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:22.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:23.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:22.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:23.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:22.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:23.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:22.746+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:23.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:22.747+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:23.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:23.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:23.143+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:23.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:23.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:23.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:23.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:23.796 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:23.503+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:23.797 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:23.504+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:24.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:23.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:24.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:23.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:24.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:23.794+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:24.167 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:23.795+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:24.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:24.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:24.164+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:24.165+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:24.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:24.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:24.798 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:24.487+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:24.799 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:24.488+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:24.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:24.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:24.832+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:25.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:24.833+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:25.472 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:25.212+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:25.472 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:25.212+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:25.473 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:25.212+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:25.473 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:25.212+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:25.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:25.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:25.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:25.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:25.839 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:25.471+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:25.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:25.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:26.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:25.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:26.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:25.837+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:26.522 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:26.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:26.522 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:26.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:26.522 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:26.200+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:26.522 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:26.201+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:26.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:26.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:26.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:26.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:26.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:26.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:26.794 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:26.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:26.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:26.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:26.792+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:27.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:26.793+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:27.513 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:27.410+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 23997 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:27.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:27.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:27.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:27.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:27.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:27.249+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:27.513 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:27.250+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:27.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:27.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:27.824 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:27.510+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:27.825 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:27.511+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:27.822+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:28.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:27.822+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:27.822+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:28.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:27.823+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:28.541 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:28.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:28.542 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:28.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:28.542 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:28.288+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:28.542 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:28.289+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:28.853 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:28.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:28.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:28.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:28.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:28.539+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:28.854 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:28.540+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:29.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:28.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:29.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:28.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:29.238 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:28.851+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:29.239 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:28.851+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:29.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:29.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:29.568 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:29.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:29.569 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:29.276+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:29.569 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:29.277+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:29.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:29.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:29.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:29.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:29.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:29.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:29.838 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:29.567+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:29.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:29.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:29.836+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:29.837+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:30.581 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:30.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:30.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:30.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:30.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:30.280+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:30.582 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:30.281+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:30.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:30.807+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:30.808+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:30.808+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:30.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:30.808+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:30.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:30.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:30.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:30.580+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:31.542 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:31.246+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:31.542 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:31.246+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:31.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:31.246+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:31.543 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:31.247+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:31.832 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:31.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:31.832 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:31.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:31.832 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:31.540+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:31.833 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:31.541+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:32.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:32.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:32.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:31.830+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:32.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:31.831+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:32.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:32.413+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24089 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:32.549 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:32.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:32.549 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:32.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:32.549 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:32.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:32.549 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:32.296+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:32.550 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:32.297+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:32.828 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:32.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:32.828 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:32.546+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:32.829 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:32.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:32.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:32.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:32.826+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:33.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:32.827+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:33.616 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:33.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:33.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:33.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:33.568+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:33.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:33.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:33.617 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:33.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:33.618 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:33.339+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:33.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:33.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:33.867+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:34.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:33.867+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:34.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:34.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:34.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:34.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:34.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:34.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:34.593+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:34.593+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:34.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:34.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:34.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:35.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:34.897+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:35.627 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:35.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:35.628 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:35.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:35.628 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:35.353+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:35.628 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:35.354+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:35.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:35.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:35.625+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:35.626+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:36.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:35.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:35.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:35.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.340 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:35.935+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:36.668 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:36.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:36.668 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:36.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:36.668 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:36.338+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:36.669 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:36.339+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:36.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:36.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:36.901+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:36.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:36.902+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:36.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:36.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:36.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:36.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:36.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:36.665+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:36.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:36.666+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:37.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:37.416+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24188 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:37.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:37.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:37.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:37.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:37.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:37.354+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:37.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:37.355+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:37.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:37.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:37.923+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:37.924+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:37.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:37.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:37.680+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:37.681+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:38.662 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:38.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:38.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:38.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:38.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:38.397+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:38.663 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:38.398+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:38.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:38.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:38.896+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:38.896+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:38.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:38.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:38.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:38.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:38.661+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:39.705 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:39.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:39.706 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:39.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:39.706 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:39.427+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:39.706 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:39.427+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:39.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:39.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:39.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:40.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:39.939+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:40.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:39.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:39.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:39.703+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:39.704+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:40.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:40.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:40.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:40.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:40.674+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:40.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:40.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:40.457+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:40.458+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:40.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:40.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:40.950+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:40.951+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:41.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:41.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:41.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:41.642+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:41.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:41.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:41.486+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:41.487+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:42.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:41.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:42.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:41.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:42.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:41.935+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:42.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:41.935+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:42.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:42.419+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24282 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:42.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:42.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:42.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:42.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:42.474+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:42.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:42.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:42.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:42.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:42.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:42.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:42.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:42.616+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:42.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:42.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:42.980+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:42.980+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:43.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:43.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:43.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:43.647+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:43.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:43.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:43.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:43.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:43.511+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:43.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:43.512+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:43.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:43.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:43.981+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:43.982+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:44.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:44.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:44.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:44.680+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:44.503+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:44.503+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:44.503+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:44.504+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:44.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:44.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:44.951+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:44.952+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:45.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:45.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:45.701+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:45.701+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:45.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:45.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:45.480+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:45.480+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:46.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:45.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:46.302 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:45.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:46.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:45.984+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:46.303 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:45.984+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:46.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:46.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:46.437+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:46.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:46.438+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:47.034 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:46.717+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:47.034 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:46.717+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:47.034 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:46.717+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:47.034 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:46.718+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:47.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:47.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:47.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:47.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:47.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:47.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:47.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:47.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:47.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:47.423+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24378 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:47.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:47.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:47.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:47.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:47.412+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:47.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:47.413+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:48.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:47.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:48.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:47.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:48.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:47.729+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:48.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:47.730+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:48.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:48.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:48.020+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:48.020+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:48.776 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:48.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:48.776 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:48.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:48.776 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:48.452+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:48.776 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:48.453+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:49.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.001+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:49.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:48.775+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:49.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:48.775+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:49.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:48.775+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:49.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:48.776+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:49.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:49.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:49.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:49.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:49.783 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:49.470+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:49.784 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:49.471+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:50.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:50.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:50.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.972+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:50.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:49.973+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:50.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:49.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:50.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:49.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:50.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:49.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:50.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:49.782+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:50.820 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:50.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:50.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:50.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:50.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:50.451+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:50.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:50.452+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.001+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:51.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:50.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:50.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:50.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:51.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:50.819+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:51.810 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:51.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:51.810 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:51.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:51.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:51.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:51.811 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:51.486+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:52.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:51.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:51.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:51.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:51.808+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:52.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:51.809+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:52.802 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:52.424+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24471 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:52.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:52.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:52.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:52.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:52.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:52.483+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:52.802 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:52.484+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:53.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:53.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:53.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:53.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:53.044+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:52.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:52.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:52.799+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:53.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:52.800+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:53.756 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:53.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:53.756 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:53.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:53.756 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:53.454+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:53.756 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:53.455+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:54.084 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:53.754+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:54.084 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:53.754+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:54.084 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:53.754+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:54.084 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:53.754+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:54.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:54.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:54.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:54.082+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:54.735 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:54.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:54.736 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:54.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:54.736 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:54.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:54.736 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:54.482+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:55.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:54.733+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:55.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:54.733+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:55.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:54.733+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:55.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:54.734+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:55.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:55.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:55.127+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:55.128+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:55.734 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:55.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:55.734 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:55.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:55.734 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:55.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:55.734 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:55.476+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:56.112 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:55.732+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:56.112 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:55.732+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:56.113 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:55.732+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:56.113 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:55.733+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:56.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:56.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:56.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:56.111+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:56.766 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:56.514+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:56.767 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:56.514+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:56.767 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:56.514+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:56.767 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:56.514+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:57.086 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:56.764+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:57.087 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:56.764+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:57.087 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:56.764+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:57.087 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:56.765+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:57.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:57.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:57.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:57.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:57.759 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:52:57.427+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24569 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:52:57.759 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:57.477+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:57.760 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:57.477+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:57.760 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:57.477+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:57.760 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:57.478+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:58.111 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:57.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.112 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:57.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.112 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:57.757+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.112 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:57.758+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:58.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:58.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:58.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:58.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:58.493+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:58.493+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:58.493+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:58.493+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:58.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:58.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:58.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:58.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:52:58.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:58.722+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:59.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:59.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:59.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:52:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:52:59.101+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:52:59.731 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:59.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:59.732 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:59.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:59.732 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:59.482+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:52:59.732 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:52:59.483+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:00.141 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:59.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:00.141 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:59.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:00.142 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:59.730+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:00.142 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:52:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:52:59.731+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:00.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:00.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:00.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:00.140+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:00.781 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:00.510+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:00.781 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:00.510+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:00.781 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:00.510+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:00.781 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:00.511+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:01.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:00.779+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:01.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:00.779+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:01.153 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:00.779+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:01.154 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:00.779+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:01.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:01.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:01.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:01.152+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:01.820 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:01.542+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:01.820 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:01.542+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:01.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:01.542+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:01.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:01.542+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:02.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:01.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:02.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:01.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:02.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:01.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:02.175 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:01.819+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:02.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:02.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:02.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:02.172+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:02.814 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:02.430+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24674 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:02.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:02.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:02.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:02.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:02.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:02.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:02.815 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:02.565+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:02.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:02.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:02.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:02.813+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:03.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:03.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:03.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:03.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:03.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:03.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:03.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:03.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:03.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:03.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:03.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:03.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:03.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:03.564+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:03.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:03.565+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:03.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:03.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:03.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:03.818+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:04.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:04.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:04.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:04.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:04.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:04.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:04.578 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:04.187+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:04.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:04.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:04.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:04.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:04.781+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:04.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:04.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:04.575+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:04.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:04.576+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:05.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:05.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:05.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:05.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:05.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:05.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:05.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:05.189+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:05.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:05.780+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:05.781+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:05.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:05.781+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:05.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:05.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:05.566+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:05.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:05.567+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:06.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:06.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:06.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:06.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:06.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:06.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:06.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:06.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:06.798+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:06.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:06.799+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:06.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:06.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:06.583+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:06.584+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:07.537 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:07.433+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24770 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:07.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:07.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:07.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:07.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:07.230+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:07.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:07.767+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:07.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:07.767+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:07.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:07.767+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:07.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:07.768+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:07.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:07.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:07.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:07.535+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:08.510 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:08.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:08.510 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:08.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:08.510 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:08.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:08.510 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:08.220+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:08.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:08.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:08.722+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:08.722+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:08.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:08.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:08.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:08.508+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:08.509+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:09.511 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:09.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:09.511 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:09.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:09.511 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:09.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:09.511 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:09.256+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:09.509+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:09.509+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:09.509+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:09.510+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:09.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:09.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:09.702+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:09.703+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:10.543 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:10.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:10.543 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:10.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:10.544 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:10.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:10.544 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:10.232+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:10.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:10.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:10.879 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:10.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:10.879 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:10.675+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:10.879 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:10.676+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:10.880 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:10.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:10.880 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:10.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:10.880 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:10.541+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:10.880 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:10.542+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:11.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:11.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:11.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:11.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:11.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:11.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:11.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:11.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:11.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:11.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:11.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:11.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:11.547+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:11.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:11.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:11.676+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:11.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:11.677+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:12.518 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:12.436+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24865 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:12.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:12.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:12.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:12.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:12.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:12.227+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:12.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:12.228+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:12.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:12.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:12.695+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:12.696+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:12.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:12.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:12.516+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:12.517+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:13.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:13.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:13.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:13.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:13.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:13.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:13.533 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:13.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:13.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:13.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:13.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:13.531+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:13.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:13.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:13.679+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:13.679+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:14.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:14.214+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:14.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:14.214+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:14.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:14.214+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:14.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:14.215+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:14.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:14.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:14.634+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:14.635+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:14.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:14.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:14.507+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:14.507+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:15.487 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:15.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:15.488 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:15.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:15.488 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:15.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:15.488 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:15.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:15.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:15.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:15.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:15.600+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:15.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:15.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:15.485+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:15.486+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:16.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:16.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:16.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:16.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:16.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:16.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:16.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:16.221+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:16.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:16.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:16.639+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:16.640+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:16.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:16.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:16.536+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:16.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:16.536+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:17.463 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:17.439+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 24963 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:17.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:17.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:17.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:17.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:17.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:17.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:17.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:17.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:17.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:17.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:17.661+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:17.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:17.662+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:17.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:17.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:17.502+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:17.503+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:18.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:18.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:18.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:18.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:18.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:18.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:18.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:18.176+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:18.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:18.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:18.646+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:18.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:18.646+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:18.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:18.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:18.535+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:18.536+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:19.559 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:19.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:19.560 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:19.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:19.560 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:19.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:19.560 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:19.191+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:19.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:19.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:19.606+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:19.607+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:19.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:19.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:19.557+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:19.558+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:20.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:20.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:20.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:20.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:20.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:20.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:20.568 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:20.187+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:20.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:20.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:20.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:20.581+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:20.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:20.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:20.566+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:20.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:20.567+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:21.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:21.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:21.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:21.174+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:21.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:21.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:21.605+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:21.606+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:21.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:21.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:21.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:21.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:22.455 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:22.442+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25058 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:22.456 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:22.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:22.456 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:22.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:22.456 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:22.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:22.456 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:22.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:22.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:22.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:22.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:22.611+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:22.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:22.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:22.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:22.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:22.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:22.583+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:22.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:22.584+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:23.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:23.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:23.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:23.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:23.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:23.231+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:23.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:23.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:23.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:23.583+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:23.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:23.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:23.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:23.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:23.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:23.570+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:23.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:23.571+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:24.610 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:24.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:24.610 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:24.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:24.610 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:24.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:24.610 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:24.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:24.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:24.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:24.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:24.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:24.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:24.618+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:24.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:24.618+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:24.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:24.608+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:24.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:24.608+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:24.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:24.608+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:24.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:24.609+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:25.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:25.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:25.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:25.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:25.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:25.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:25.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:25.273+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:25.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:25.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:25.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:25.589+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:25.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:25.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:25.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:25.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:25.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:25.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:25.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:25.575+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:26.546 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:26.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:26.546 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:26.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:26.546 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:26.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:26.546 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:26.302+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:26.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:26.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:26.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:26.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:26.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:26.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:26.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:26.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:26.545+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:27.601 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:27.445+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25153 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:27.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:27.601 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:27.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:27.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:27.319+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:27.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:27.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:27.531+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:27.602 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:27.533+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:27.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:27.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:27.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:27.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:27.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:27.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:27.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:27.599+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:28.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:28.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:28.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:28.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:28.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:28.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:28.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:28.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:28.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:28.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:28.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:28.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:28.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:28.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:28.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:28.362+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:28.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:28.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:28.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:28.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:28.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:28.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:28.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:28.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:29.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:29.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:29.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:29.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:29.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:29.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:29.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:29.386+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:29.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:29.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:29.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:29.619+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:29.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:29.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:29.562+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:29.563+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:30.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:30.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:30.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:30.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:30.545 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:30.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:30.546 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:30.384+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:30.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:30.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:30.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:30.544+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:30.544+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:30.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:30.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:30.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:30.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:30.581+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:31.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:31.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:31.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:31.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:31.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:31.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:31.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:31.574+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:31.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:31.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:31.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:31.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:31.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:31.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:31.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:31.590+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:32.565 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:32.448+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25248 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:32.565 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:32.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:32.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:32.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:32.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:32.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:32.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:32.310+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:32.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:32.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:32.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:32.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:32.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:32.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:32.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:32.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:32.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:32.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:32.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:32.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:32.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:32.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:32.639+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:33.593 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:33.280+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:33.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:33.280+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:33.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:33.280+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:33.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:33.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:33.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:33.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:33.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:33.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:33.595 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:33.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:33.595 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:33.520+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:33.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:33.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:33.592+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:33.593+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:34.529 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:34.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:34.529 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:34.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:34.529 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:34.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:34.529 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:34.264+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:34.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:34.528+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:34.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:34.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:34.544+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:34.544+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:35.547 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:35.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:35.548 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:35.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:35.548 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:35.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:35.548 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:35.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:35.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:35.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:35.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:35.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:35.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:35.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:35.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:35.254+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:35.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:35.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:35.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:35.546+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:36.528 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:36.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:36.528 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:36.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:36.528 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:36.241+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:36.528 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:36.242+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:36.529 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:36.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:36.529 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:36.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:36.529 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:36.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:36.529 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:36.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:36.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:36.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:36.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:36.526+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:37.464 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:37.450+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25345 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:37.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:37.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:37.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:37.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:37.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:37.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:37.465 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:37.211+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:37.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:37.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:37.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:37.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:37.489+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:37.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:37.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:37.545+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:37.545+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:38.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:38.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:38.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:38.164+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:38.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:38.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:38.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:38.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:38.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:38.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:38.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:38.582+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:39.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:39.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:39.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:39.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:39.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:39.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:39.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:39.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:39.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:39.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:39.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:39.603+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:40.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:40.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:40.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:40.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:40.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:40.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:40.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:40.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:40.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:40.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:40.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:40.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:40.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:40.445+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:40.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:40.446+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:41.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:41.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:41.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:41.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:41.730 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:41.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:41.730 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:41.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:41.730 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:41.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:41.730 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:41.599+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:41.731 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:41.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:41.731 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:41.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:41.731 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:41.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:41.731 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:41.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:42.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:42.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:42.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:42.135+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:42.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:42.453+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25439 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:42.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:42.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:42.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:42.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:42.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:42.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:42.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:42.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:42.634+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:43.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:43.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:43.146+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:43.147+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:43.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:43.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:43.476+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:43.477+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:43.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:43.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:43.638+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:43.638+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:44.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:44.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:44.164+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:44.165+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:44.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:44.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:44.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:44.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:44.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:44.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:44.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:44.605+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:45.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:45.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:45.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:45.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:45.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:45.189+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:45.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:45.190+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:45.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:45.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:45.450+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:45.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:45.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:45.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:45.569+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:45.570+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:46.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:46.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:46.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:46.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:46.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:46.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:46.472 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:46.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:46.470+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:46.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:46.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:46.526+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:46.527+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:47.458 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:47.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:47.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:47.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:47.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:47.205+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:47.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:47.205+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:47.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:47.456+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25534 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:47.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:47.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:47.517+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:47.518+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:47.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:47.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:47.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:47.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:47.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:48.566 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:48.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:48.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:48.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:48.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:48.224+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:48.567 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:48.224+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:48.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:48.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:48.564+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:48.565+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:48.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:48.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:48.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:48.579+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:49.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:49.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:49.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:49.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:49.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:49.209+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:49.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:49.210+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:49.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:49.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:49.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:49.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:49.558+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:49.559+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:50.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:50.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:50.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:50.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:50.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:50.210+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:50.581 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:50.212+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:50.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:50.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:50.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:50.619+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:50.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:50.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:50.579+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:50.579+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:51.615 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:51.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:51.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:51.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:51.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:51.187+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:51.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:51.188+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:51.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:51.614+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:51.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:51.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:51.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:51.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:52.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:52.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:52.165+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:52.166+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:52.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:52.459+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25628 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:52.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:52.595+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:52.596+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:52.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:52.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:52.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:52.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:52.602+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:53.571 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:53.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:53.572 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:53.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:53.572 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:53.194+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:53.572 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:53.194+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:53.886 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:53.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:53.886 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:53.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:53.886 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:53.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:53.887 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:53.570+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:53.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:53.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:53.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:53.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:53.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:53.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:53.887 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:53.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:54.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:54.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:54.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:54.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:54.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:54.203+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:54.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:54.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:54.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:54.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:54.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:54.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:54.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:54.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:54.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:54.536+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:54.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:54.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:54.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:54.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:54.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:54.617+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:54.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:54.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:55.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:55.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:55.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:55.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:55.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:55.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:55.577 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:55.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:55.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:55.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:55.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:55.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:55.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:55.600+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:55.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:55.601+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:55.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:55.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:55.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:55.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:55.574+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:55.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:55.575+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:56.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:56.617 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:56.617 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:56.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:56.617 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:56.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:56.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:56.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:56.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:56.613+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:56.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:56.614+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:56.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:56.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:56.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:56.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:56.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:56.642+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:57.587 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:53:57.462+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25725 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:53:57.588 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:57.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:57.588 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:57.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:57.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:57.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:57.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:57.274+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:57.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:57.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:57.615+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:57.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:57.616+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:57.661+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:57.661+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:57.661+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:57.662+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:58.569 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:58.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:58.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:58.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:58.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:58.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:58.570 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:58.263+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:58.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:58.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:58.567+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:58.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:58.568+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:58.655+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:58.655+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:58.655+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:58.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:58.656+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:59.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:59.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:59.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:59.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:59.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:59.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:53:59.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:53:59.303+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:59.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:59.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:59.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:53:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:53:59.537+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:53:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:59.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:59.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:59.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:53:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:53:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:53:59.664+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:00.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:00.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:00.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:00.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:00.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:00.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:00.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:00.317+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:00.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:00.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:00.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:00.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:00.677+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:00.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:00.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:00.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:00.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:00.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:00.536+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:01.583 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:01.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:01.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:01.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:01.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:01.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:01.584 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:01.273+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:01.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:01.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:01.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:01.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:01.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:01.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:01.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:01.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:01.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:01.581+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:01.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:01.582+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:02.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:02.465+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25820 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:02.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:02.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:02.592 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:02.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:02.592 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:02.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:02.592 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:02.310+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:02.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:02.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:02.604+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:02.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:02.605+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:02.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:02.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:02.647+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:02.648+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:03.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:03.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:03.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:03.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:03.349+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:03.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:03.350+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:03.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:03.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:03.641+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:03.641+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:03.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:03.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:03.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:03.677+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:04.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:04.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:04.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:04.628+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:04.629+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:04.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:04.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:04.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:04.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:04.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:04.391+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:04.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:04.392+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:04.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:04.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:04.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:04.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:04.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:04.617+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:04.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:04.618+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:05.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:05.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:05.599+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:05.600+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:05.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:05.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:05.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:05.405+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:05.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:05.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:05.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:05.640+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:05.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:05.640+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:06.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:06.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:06.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:06.391+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:06.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:06.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:06.597+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:06.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:06.597+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:06.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:06.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:06.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:06.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:06.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:06.648+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:06.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:06.649+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:07.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:07.468+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 25916 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:07.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:07.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:07.415+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:07.416+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:07.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:07.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:07.605+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:07.606+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:07.651+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:07.651+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:07.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:07.651+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:07.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:07.652+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:08.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:08.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:08.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:08.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:08.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:08.376+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:08.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:08.377+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:08.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:08.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:08.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:08.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:08.665+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:08.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:08.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:08.651+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:08.652+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:09.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:09.356+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:09.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:09.356+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:09.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:09.356+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:09.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:09.357+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:09.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:09.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:09.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:09.664+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:09.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:09.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:09.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:09.699+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:09.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:09.699+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:10.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:10.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:10.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:10.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:10.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:10.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:10.331+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:10.721+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:10.721+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:10.721+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:11.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:10.721+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:10.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:10.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:10.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:10.703+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:11.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:11.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:11.293+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:11.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:11.293+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:11.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:11.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:11.674+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:11.675+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:11.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:11.746+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:12.597 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:12.471+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26011 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:12.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:12.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:12.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:12.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:12.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:12.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:12.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:12.279+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:12.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:12.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:12.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:12.775+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:12.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:12.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:12.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:12.685+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:12.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:12.686+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:13.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:13.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:13.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:13.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:13.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:13.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:13.623 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:13.264+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:13.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:13.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:13.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:13.660+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:13.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:13.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:13.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:13.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:13.808+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:14.649 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:14.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:14.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:14.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:14.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:14.294+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:14.650 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:14.294+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:14.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:14.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:14.647+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:14.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:14.648+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:14.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:14.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:14.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:14.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:14.848+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:15.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:15.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:15.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:15.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:15.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:15.339+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:15.666 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:15.339+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:15.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:15.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:15.664+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:15.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:15.664+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:15.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:15.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:15.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:15.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:15.887+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:16.670 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:16.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:16.671 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:16.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:16.671 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:16.383+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:16.671 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:16.384+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:16.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:16.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:16.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:16.844+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:16.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:16.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:16.668+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:16.669+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:17.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:17.476+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26109 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:17.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:17.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:17.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:17.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:17.415+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:17.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:17.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:17.633+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:17.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:17.634+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:17.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:17.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:17.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:17.805+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:18.630+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:18.630+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:18.630+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:18.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:18.631+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:18.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:18.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:18.385+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:18.386+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:18.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:18.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:18.813+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:18.814+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:19.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:19.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:19.594+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:19.679 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:19.595+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:19.680 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:19.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:19.680 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:19.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:19.680 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:19.367+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:19.680 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:19.368+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:19.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:19.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:19.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:19.853+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:20.547 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:20.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:20.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:20.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:20.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:20.411+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:20.548 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:20.412+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:20.811 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:20.545+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:20.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:20.545+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:20.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:20.545+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:20.812 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:20.546+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:20.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:20.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:20.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:20.810+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:21.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:21.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:21.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:21.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:21.389+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:21.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:21.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:21.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:21.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:21.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:21.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:21.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:21.797+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:22.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:22.478+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26204 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:22.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:22.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:22.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:22.519+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:22.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:22.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:22.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:22.360+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:22.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:22.361+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:22.782+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:22.782+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:22.782+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:22.783+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:23.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:23.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:23.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:23.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:23.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:23.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:23.625 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:23.363+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:23.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:23.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:23.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:23.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:23.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:23.526+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:23.626 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:23.527+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:23.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:23.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:23.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:23.826+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:24.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:24.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:24.389+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:24.389+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:24.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:24.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:24.571+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:24.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:24.573+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:24.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:24.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:24.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:24.826+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:25.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:25.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:25.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:25.566+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:25.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:25.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:25.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:25.424+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:25.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:25.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:25.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:25.857+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:26.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:26.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:26.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:26.615+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:26.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:26.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:26.387+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:26.387+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:26.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:26.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:26.807+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:26.808+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:27.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:27.481+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26300 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:27.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:27.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:27.600+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:27.600+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:27.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:27.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:27.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:27.678 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:27.419+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:27.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:27.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:27.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:27.802+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:28.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:28.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:28.582+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:28.582+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:28.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:28.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:28.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:28.395+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:28.791+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:28.791+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:28.791+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:28.792+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:29.563 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:29.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:29.563 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:29.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:29.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:29.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:29.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:29.363+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:29.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:29.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:29.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:29.785+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:29.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:29.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:29.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:29.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:29.562+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:30.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:30.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:30.538 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:30.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:30.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:30.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:30.539 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:30.344+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:30.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:30.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:30.792 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:30.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:30.793 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:30.536+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:30.793 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:30.537+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:30.791+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:31.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:31.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:31.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:31.568+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:31.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:31.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:31.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:31.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:31.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:31.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:31.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:31.343+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:31.777+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:31.777+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:31.777+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:31.778+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:32.590 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:32.484+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26395 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:32.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:32.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:32.590 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:32.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:32.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:32.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:32.591 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:32.319+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:32.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:32.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:32.588+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:32.589+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:32.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:32.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:32.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:32.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:32.763+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:33.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:33.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:33.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:33.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:33.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:33.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:33.564 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:33.308+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:33.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:33.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:33.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:33.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:33.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:33.561+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:33.822 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:33.562+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:33.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:33.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:33.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:33.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:33.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:33.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:33.823 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:33.750+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:34.527+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:34.528+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:34.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:34.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:34.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:34.341+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:34.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:34.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:34.774+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:34.775+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:35.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:35.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:35.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:35.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:35.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:35.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:35.323+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:35.323+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:35.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:35.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:35.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:35.796+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:36.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:36.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:36.536 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:36.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:36.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:36.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:36.537 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:36.282+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:36.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:36.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:36.534+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:36.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:36.778+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:36.778+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:36.778+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:36.779+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:37.614 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:37.487+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26493 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:37.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:37.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:37.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:37.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:37.495+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:37.496+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:37.286+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:37.286+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:37.286+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:37.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:37.287+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:37.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:37.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:37.750+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:37.751+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:38.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:38.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:38.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:38.516+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:38.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:38.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:38.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:38.271+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:38.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:38.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:38.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:38.767+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:39.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:39.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:39.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:39.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:39.297+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:39.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:39.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:39.513+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:39.514+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:40.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:39.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:40.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:39.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:40.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:39.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:40.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:39.788+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:40.499 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:40.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:40.499 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:40.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:40.499 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:40.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:40.499 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:40.264+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:40.808 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:40.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:40.809 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:40.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:40.809 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:40.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:40.809 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:40.498+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:40.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:40.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:40.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:40.807+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:41.549 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:41.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:41.550 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:41.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:41.550 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:41.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:41.550 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:41.219+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:41.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:41.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:41.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:41.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:41.548+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:41.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:41.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:41.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:41.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:41.795+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:42.515 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:42.490+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26587 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:42.517 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:42.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:42.517 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:42.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:42.517 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:42.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:42.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:42.261+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:42.768 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:42.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:42.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:42.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:42.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:42.592+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:42.769 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:42.593+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:43.038 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:42.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.038 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:42.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.039 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:42.766+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.039 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:42.767+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:43.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:43.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:43.276+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:43.277+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:43.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:43.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:43.552+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:43.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:43.553+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:43.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:43.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:43.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:43.730+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:44.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:44.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:44.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:44.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:44.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:44.230+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:44.509 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:44.231+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:44.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:44.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:44.506+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:44.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:44.507+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:44.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:44.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:44.699+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:44.699+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:45.463 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:45.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:45.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:45.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:45.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:45.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:45.464 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:45.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:45.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:45.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:45.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:45.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:45.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:45.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:45.461+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:45.462+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:46.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:46.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:46.199+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:46.200+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:46.470+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:46.471+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:46.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:46.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:46.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:46.678+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:47.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:47.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:47.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:47.152+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:47.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:47.493+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26682 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:47.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:47.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:47.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:47.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:47.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:47.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:47.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:47.446+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:47.929 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:47.447+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:48.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:48.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:48.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:48.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:48.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:48.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:48.457+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:48.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:48.458+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:48.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:48.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:48.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:48.654+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:49.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:49.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:49.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:49.124+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:49.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:49.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:49.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:49.643+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:49.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:49.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:49.425+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:49.426+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:50.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:50.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:50.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:50.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:50.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:50.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:50.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:50.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:50.672 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:50.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:50.672 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:50.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:50.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:50.419+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:50.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:50.420+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:50.670+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:50.670+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:50.670+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:50.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:50.671+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:51.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:51.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:51.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:51.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:51.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:51.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:51.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:51.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:51.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:51.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:51.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:51.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:51.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:51.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:51.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:51.403+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:51.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:51.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:51.653+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:51.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:51.653+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:52.498 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:52.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:52.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:52.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:52.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:52.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:52.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:52.360+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:52.500 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:52.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:52.500 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:52.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:52.500 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:52.185+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:52.500 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:52.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:52.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:52.496+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26777 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:52.666+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:52.666+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:52.666+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:52.667+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:53.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:53.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:53.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:53.390+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:53.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:53.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:53.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:53.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:53.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:53.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:53.667+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:53.668+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:54.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:54.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:54.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:54.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:54.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:54.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:54.408 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:54.147+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:54.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:54.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:54.405+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:54.406+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:55.172 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:54.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:54.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:54.674+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.173 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:54.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:55.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:55.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:55.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:55.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:55.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:55.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:55.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:55.455+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:55.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:55.456+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:55.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:55.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:55.645+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:55.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:55.646+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:56.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:56.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:56.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:56.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:56.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:56.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:56.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:56.197+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:56.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:56.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:56.439+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:56.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:56.440+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:56.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:56.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:56.675+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:56.675+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:57.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:57.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:57.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:57.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:57.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:57.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:57.502 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:57.232+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:57.502 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:57.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:57.503 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:57.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:57.503 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:57.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:57.503 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:57.397+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:57.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:54:57.499+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26875 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:54:57.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:57.652+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:57.652+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:57.652+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:57.653+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:58.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:58.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:58.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:58.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:58.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:58.390+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:58.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:58.391+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:58.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:58.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:58.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:58.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:58.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:58.192+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:58.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:58.193+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:58.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:58.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:58.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:58.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:58.632+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:59.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:59.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:59.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:59.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:59.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:59.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:54:59.414 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:54:59.151+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:59.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:59.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:59.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:59.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:59.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:59.412+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:54:59.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:54:59.412+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:54:59.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:59.665+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:54:59.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:54:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:54:59.666+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:00.451 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:00.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:00.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:00.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:00.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:00.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:00.452 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:00.197+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:00.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:00.449+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:00.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:00.450+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:00.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:00.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:00.641+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:00.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:00.641+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:01.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:01.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:01.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:01.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:01.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:01.161+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:01.422 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:01.161+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:01.420+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:01.421+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:01.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:01.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:01.621+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:01.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:01.621+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:02.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:02.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:02.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:02.373+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:02.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:02.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:02.168+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:02.169+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:02.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:02.504+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 26970 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:02.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:02.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:02.590+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:02.591+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:03.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:03.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:03.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:03.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:03.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:03.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:03.599 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:03.181+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:03.600 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:03.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:03.600 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:03.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:03.600 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:03.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:03.600 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:03.370+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:03.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:03.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:03.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:03.599+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:04.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:04.149+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:04.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:04.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:04.359+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:04.360+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:04.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:04.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:04.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:04.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:04.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:04.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:04.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:04.582+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:05.617 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:05.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:05.617 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:05.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:05.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:05.177+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:05.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:05.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:05.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:05.618 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:05.393+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:05.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:05.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:05.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:05.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:05.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:06.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:06.170+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:06.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:06.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:06.393+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:06.394+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:06.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:06.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:06.602+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:06.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:06.602+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:07.396 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:07.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:07.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:07.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:07.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:07.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:07.397 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:07.135+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:07.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:07.506+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27065 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:07.394+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:07.394+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:07.394+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:07.395+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:07.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:07.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:07.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:07.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:07.592+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:08.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:08.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:08.135+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:08.136+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:08.698 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:08.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:08.698 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:08.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:08.698 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:08.429+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:08.698 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:08.430+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:08.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:08.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:08.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:08.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:08.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:08.578+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:08.699 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:08.578+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:09.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:09.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:09.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:09.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:09.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:09.144+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:09.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:09.144+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:09.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:09.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:09.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:09.409+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:09.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:09.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:09.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:09.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:10.458 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:10.458 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:10.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:10.459 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:10.180+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:10.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:10.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:10.456+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:10.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:10.457+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:10.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:10.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:10.663+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:10.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:10.664+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:11.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:11.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:11.482 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:11.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:11.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:11.211+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:11.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:11.212+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:11.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:11.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:11.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:11.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:11.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:11.480+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:11.736 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:11.481+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:11.736 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:11.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:11.737 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:11.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:11.737 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:11.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:11.737 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:11.698+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:12.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:12.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:12.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:12.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:12.512 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:12.244+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:12.513 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:12.245+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:12.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:12.509+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27160 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:12.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:12.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:12.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:12.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:12.524+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:12.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:12.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:12.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:12.650+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:13.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:13.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:13.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:13.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:13.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:13.492+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:13.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:13.492+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:13.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:13.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:13.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:13.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:13.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:13.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:13.629 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:13.274+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:13.680+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:13.680+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:13.680+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:13.681+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:14.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:14.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:14.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:14.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:14.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:14.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:14.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:14.466+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:14.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:14.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:14.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:14.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:14.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:14.233+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:14.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:14.233+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:14.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:14.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:14.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:14.632+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:15.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:15.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:15.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:15.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:15.615 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:15.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:15.616 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:15.419+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:15.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:15.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:15.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:15.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:15.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:15.184+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:15.616 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:15.185+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:15.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:15.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:15.613+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:15.614+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:16.415 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:16.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:16.415 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:16.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:16.416 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:16.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:16.416 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:16.163+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:16.416 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:16.164+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:16.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:16.413+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:16.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:16.414+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:16.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:16.594+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:16.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:16.594+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:16.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:16.594+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:16.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:16.594+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:17.515 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:17.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:17.515 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:17.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:17.515 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:17.370+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:17.515 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:17.371+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:17.515 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:17.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:17.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:17.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:17.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:17.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:17.516 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:17.191+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:17.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:17.512+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27257 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:17.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:17.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:17.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:17.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:17.631+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:18.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:18.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:18.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:18.410+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:18.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:18.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:18.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:18.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:18.186+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:18.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:18.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:18.677+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:18.678+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:19.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:19.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:19.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:19.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:19.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:19.221+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:19.636 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:19.222+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:19.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:19.380+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:19.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:19.380+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:19.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:19.380+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:19.637 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:19.381+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:19.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:19.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:19.633+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:19.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:19.633+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:20.593 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:20.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:20.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:20.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:20.269+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:20.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:20.594 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:20.361+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:20.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:20.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:20.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:20.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:20.592+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:21.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:21.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:21.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:21.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:21.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:21.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:21.585 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:21.265+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:21.586 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:21.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:21.586 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:21.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:21.586 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:21.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:21.586 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:21.359+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:21.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:21.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:21.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:21.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:21.584+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:22.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:22.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:22.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:22.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:22.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:22.389+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:22.519 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:22.390+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:22.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:22.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:22.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:22.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:22.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:22.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:22.519 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:22.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:22.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:22.516+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27351 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:22.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:22.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:22.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:22.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:22.596+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:23.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:23.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:23.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:23.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:23.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:23.173+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:23.344 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:23.173+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:23.628 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:23.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:23.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:23.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:23.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:23.341+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:23.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:23.342+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:23.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:23.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:23.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:23.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:24.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:24.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:24.597 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:24.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:24.181+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:24.182+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:24.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:24.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:24.334+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:24.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:24.335+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:24.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:24.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:24.595+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:24.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:24.596+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:25.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:25.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:25.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:25.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:25.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:25.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:25.561 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:25.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:25.561 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:25.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:25.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:25.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:25.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:25.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:25.562 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:25.342+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:25.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:25.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:25.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:25.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:26.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:26.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:26.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:26.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:26.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:26.186+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:26.334 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:26.187+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:26.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:26.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:26.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:26.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:26.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:26.331+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:26.606 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:26.332+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:26.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:26.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:26.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:26.603+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:26.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:26.604+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:27.354+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:27.354+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:27.354+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:27.354+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:27.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:27.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:27.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:27.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:27.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:27.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:27.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:27.142+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:27.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:27.519+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27447 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:27.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:27.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:27.570+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:27.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:27.571+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:28.399 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:28.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:28.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:28.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:28.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:28.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:28.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:28.106+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:28.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:28.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:28.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:28.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:28.589+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:28.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:28.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:28.397+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:28.398+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:29.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:29.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:29.399 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:29.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:29.399 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:29.061+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:29.399 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:29.062+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:29.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:29.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:29.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:29.397+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:29.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:29.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:29.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:29.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:29.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:29.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:29.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:29.583+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:30.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:30.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:30.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:30.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:30.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:30.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:30.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:30.062+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:30.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:30.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:30.587+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:30.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:30.588+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:30.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:30.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:30.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:30.379+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:31.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:31.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:31.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:31.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:31.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:31.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:31.046+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:31.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:31.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:31.577+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:31.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:31.578+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:31.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:31.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:31.422+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:31.423+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:32.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:32.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:32.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:32.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:32.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:32.035+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:32.413 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:32.036+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:32.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:32.522+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27542 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:32.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:32.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:32.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:32.410+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:32.411+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:32.540+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:32.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:32.542+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:33.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:33.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:33.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:33.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:33.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:33.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:33.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:33.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:33.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:33.368+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:33.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:33.368+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:33.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:33.368+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:33.629 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:33.369+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:33.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:33.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:33.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:33.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:33.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:33.584+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:33.630 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:33.585+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:34.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:34.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:34.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:34.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:34.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:34.026+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:34.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:34.026+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:34.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:34.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:34.582+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:34.582+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:34.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:34.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:35.420 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:35.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:35.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:35.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:35.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:35.066+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:35.421 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:35.067+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:35.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:35.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:35.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:35.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:35.419+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:35.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:35.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:35.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:35.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:35.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:35.579+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:35.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:35.580+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:36.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:36.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:36.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:36.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:36.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:36.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:36.454+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:36.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:36.455+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:36.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:36.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:36.616+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:36.618+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:37.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:37.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:37.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:37.079+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:37.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:37.525+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27640 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:37.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:37.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:37.601+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:37.602+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:37.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:37.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:37.482+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:37.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:37.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:38.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:38.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:38.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:38.057+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:38.508+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:38.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:38.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:38.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:38.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:38.552+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:38.552+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:39.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:39.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:39.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:39.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:39.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:39.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:39.522+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:39.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:39.522+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:39.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:39.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:39.558+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:39.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:39.559+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:40.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:40.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:40.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:40.092+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:40.543+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:40.543+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:40.543+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:40.544+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:40.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:40.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:40.509+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:40.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:40.510+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:41.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:41.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:41.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:41.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:41.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:41.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:41.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:41.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:41.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:41.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:41.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:41.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:41.475+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:41.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:41.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:41.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:41.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:41.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:41.591+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:41.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:41.592+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:42.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:42.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:42.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:42.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:42.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:42.529+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27734 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:42.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:42.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:42.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:42.588+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:42.589+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:42.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:42.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:42.514+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:42.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:42.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:43.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:43.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:43.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:43.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:43.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:43.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:43.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:43.634+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:43.635+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:43.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:43.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:43.565+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:43.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:43.566+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:44.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:44.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:44.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:44.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:44.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:44.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:44.649+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:44.650+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:44.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:44.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:44.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:44.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:44.578+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:44.928 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:44.579+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:45.152+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:45.152+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:45.152+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:45.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:45.153+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:45.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:45.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:45.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:45.551+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:45.552+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:45.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:45.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:45.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:45.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:46.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:46.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:46.112+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:46.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:46.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:46.589+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:46.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:46.589+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:46.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:46.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:46.673+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:46.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:46.674+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:47.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:47.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:47.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:47.089+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:47.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:47.533+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27829 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:47.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:47.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:47.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:47.614+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:47.614+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:47.690+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:47.690+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:47.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:47.690+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:47.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:47.691+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:48.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:48.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:48.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:48.054+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:48.741+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:48.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:48.741+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:48.741+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:48.742+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:48.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:48.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:48.599+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:48.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:48.599+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:49.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:49.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:49.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:49.040+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:49.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:49.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:49.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:49.768+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:49.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:49.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:49.586+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:49.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:49.587+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.019+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.020+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:50.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:50.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:50.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:50.585+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:50.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:50.586+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:50.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:50.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:50.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:50.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:50.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:50.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:50.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:50.805+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:50.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:51.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:51.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:51.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:51.619+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:51.619+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:51.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:51.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:51.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:51.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:51.841+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:52.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:52.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:52.028+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:52.029+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:52.800 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:52.535+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 27924 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:52.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:52.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:52.800 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:52.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:52.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:52.660+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:52.801 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:52.661+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:53.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:52.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:52.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:52.798+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:52.799+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:53.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:53.000+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:53.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:53.000+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:53.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:53.000+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:53.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:53.001+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:53.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:53.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:53.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:53.755+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:53.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:53.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:53.711+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:53.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:53.712+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:54.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:54.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:54.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:54.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:54.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:54.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:54.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:54.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:54.788+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:54.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:54.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:54.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:54.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:54.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:54.672+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:54.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:54.673+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:55.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:55.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:55.010+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:55.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:55.011+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:56.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:55.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:56.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:55.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:56.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:55.815+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:56.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:55.816+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:56.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:55.704+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:56.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:55.704+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:56.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:55.704+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:56.021 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:55.705+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.019+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:57.001 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:56.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.002 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:56.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.002 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:56.713+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.002 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:56.714+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:57.002 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:56.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.002 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:56.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:56.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.003 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:56.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:57.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:56.999+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:57.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:57.000+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:57.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:55:57.539+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28021 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:57.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:57.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:57.696+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:57.697+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:57.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:57.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:57.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:57.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:57.746+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:58.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:58.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:58.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:58.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:58.019+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:59.018 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:58.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:59.018 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:58.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:59.018 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:58.697+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:55:59.019 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:58.698+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:59.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:58.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:59.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:58.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:59.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:58.736+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:55:59.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:58.737+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:55:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:59.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:59.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:59.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:55:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:55:59.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:00.060 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:59.737+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:00.061 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:59.737+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:00.061 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:59.737+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:00.061 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:55:59.739+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:00.061 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:59.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:00.061 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:59.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:00.062 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:59.755+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:00.062 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:55:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:55:59.756+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:00.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:00.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:00.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:00.059+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:01.064 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:00.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:00.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:00.763+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:00.765+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:00.773+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:00.773+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:01.065 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:00.773+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:01.066 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:00.774+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:01.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:01.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:01.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:01.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:02.058 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:01.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:02.058 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:01.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:02.058 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:01.812+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:02.058 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:01.813+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:02.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:01.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:01.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:01.728+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:01.729+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:02.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:02.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:02.055+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:02.056+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:02.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:02.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:02.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:02.712+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:02.813 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:02.713+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:02.813 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:02.542+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28115 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:02.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:02.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:02.810+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:02.811+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:03.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:03.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:03.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:03.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:03.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:03.012+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:03.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:03.013+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:03.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:03.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:03.838+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:03.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:03.839+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:03.684+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:03.684+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:03.684+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:03.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:03.685+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:04.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:04.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:04.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:04.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:04.004+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:04.483 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:04.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:05.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:04.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:04.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:04.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:04.791+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:05.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:04.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:04.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:04.702+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.008 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:04.703+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:05.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:05.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:05.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:05.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:05.982 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:05.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.982 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:05.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:05.817+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:05.817+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:05.698+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:05.698+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:05.698+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:05.983 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:05.699+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.038+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:06.992 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:06.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:06.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:06.803+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:06.803+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:06.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:06.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:06.993 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:06.714+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:06.994 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:06.715+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.990+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:06.991+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:07.818 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:07.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:07.818 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:07.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:07.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:07.749+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:07.819 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:07.749+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:07.819 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:07.545+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28212 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:07.816+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:07.816+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:08.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:07.816+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:08.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:07.817+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:08.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:08.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:08.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:08.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:08.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:09.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:08.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:09.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:08.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:08.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:08.819+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:08.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:08.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:08.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:09.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:08.785+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:09.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:09.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:09.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:09.025+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:10.070 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:09.864+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:10.071 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:09.864+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:10.071 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:09.864+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:10.071 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:09.864+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:10.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:09.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:10.071 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:09.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:10.072 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:09.770+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:10.072 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:09.771+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:10.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:11.096 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:10.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:11.096 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:10.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:11.096 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:10.904+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:11.097 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:10.904+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:11.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:10.776+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:11.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:10.776+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:11.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:10.776+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:11.097 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:10.777+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:11.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:11.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:11.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:11.095+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:12.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:11.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:11.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:11.784+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:11.785+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:11.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:11.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:12.128 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:11.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:12.129 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:11.897+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:12.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:12.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:12.130+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:12.131+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:12.828 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:12.549+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28305 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:13.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:12.870+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:12.870+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:12.870+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.126 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:12.871+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:13.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:12.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:12.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:12.826+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:12.827+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:13.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:13.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:13.124+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:13.125+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:13.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:13.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:13.844+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:13.844+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:13.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:13.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:13.827+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:13.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:13.828+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:14.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:14.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:14.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:14.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:15.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:14.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:15.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:14.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:15.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:14.811+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:15.150 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:14.812+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:15.150 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:14.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:15.151 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:14.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:15.151 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:14.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:15.151 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:14.852+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:15.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:15.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:15.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:15.148+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:15.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:15.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:15.771+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:16.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:15.772+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:15.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:15.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:15.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:16.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:15.868+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:16.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:16.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:16.197+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:16.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:16.197+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:16.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:16.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:16.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:16.867+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:16.740+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:16.740+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:16.740+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:16.740+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:17.554 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:17.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:17.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:17.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:17.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:17.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:17.555 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:17.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:17.879 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:17.552+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28404 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:17.879 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:17.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:17.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:17.790+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:17.880 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:17.791+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:17.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:17.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:17.877+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:17.878+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:18.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:18.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:18.219+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:18.220+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:18.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:18.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:18.880+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:18.881+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:18.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:18.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:18.818+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:19.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:18.818+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:19.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:19.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:19.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:19.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:19.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:19.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:19.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:19.230+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:19.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:19.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:19.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:19.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:20.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:19.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:20.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:19.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:20.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:19.857+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:20.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:19.857+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:20.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:20.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:20.190+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:20.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:20.191+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:20.905+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:20.905+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:21.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:20.905+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:21.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:20.906+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:20.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:20.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:20.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:20.922+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:21.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:21.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:21.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:21.215+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:22.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:21.889+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:22.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:21.889+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:22.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:21.889+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:22.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:21.890+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:22.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:21.895+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:22.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:21.895+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:22.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:21.895+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:22.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:21.895+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:22.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:22.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:22.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:22.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:22.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:22.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:22.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:22.251+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:22.864 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:22.555+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28498 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:22.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:22.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:22.862+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:23.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:22.863+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:22.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:22.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:22.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:22.893+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:23.633 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:23.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:23.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:23.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:23.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:23.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:23.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:23.278+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:23.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:23.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:23.942+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:24.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:23.943+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:24.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:23.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:24.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:23.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:24.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:23.884+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:24.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:23.885+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:24.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:24.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:24.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:24.321+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:24.919+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:24.919+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:24.919+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:25.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:24.920+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:24.917+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:24.917+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:24.917+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:25.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:24.918+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:25.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:25.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:25.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:25.325+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:26.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:25.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:26.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:25.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:26.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:25.896+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:26.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:25.897+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:26.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:25.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:26.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:25.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:26.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:25.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:26.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:25.922+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:26.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:26.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:26.319+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:26.321+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:27.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:26.944+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:26.944+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:26.944+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:26.945+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:26.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:26.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:26.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:27.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:26.909+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:27.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:27.559+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28595 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:27.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:27.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:27.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:27.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:27.370+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:27.946 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:27.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.947 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:27.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.947 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:27.920+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:27.947 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:27.921+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:27.944+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:27.944+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:27.944+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:28.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:27.945+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:28.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:28.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:28.370+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:28.370+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:28.915+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:28.915+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:28.915+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:29.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:28.916+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:28.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:28.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:28.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:29.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:28.937+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:29.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:29.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:29.342+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:29.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:30.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:29.924+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:30.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:29.924+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:30.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:29.924+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:30.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:29.924+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:30.213 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:29.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:30.213 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:29.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:30.213 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:29.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:30.213 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:29.985+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:30.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:30.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:30.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:30.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:30.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:30.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:30.558 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:30.322+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:31.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:30.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:31.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:30.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:31.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:30.953+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:31.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:30.954+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:31.373 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:31.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:31.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:31.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:31.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:31.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:31.374 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:31.031+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:31.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:31.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:31.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:31.372+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:32.007 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:31.945+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:32.008 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:31.945+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:32.008 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:31.945+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:32.008 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:31.946+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:32.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:32.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:32.373 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:32.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:32.373 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:32.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:32.373 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:32.006+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:32.676 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:32.563+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28689 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:32.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:32.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:32.371+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:32.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:32.372+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:33.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:32.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:33.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:32.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:33.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:32.996+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:33.351 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:32.997+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:33.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:33.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:33.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:33.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:33.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:33.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:33.352 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:33.049+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:33.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:33.350+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:33.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:33.350+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:33.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:33.350+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:33.630 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:33.351+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:34.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:34.044+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:34.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:34.044+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:34.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:34.044+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:34.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:34.045+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:34.368 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:34.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:34.368 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:34.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:34.368 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:34.031+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:34.368 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:34.031+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:34.366+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:34.366+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:34.366+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:34.366+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:35.414 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:35.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:35.414 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:35.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:35.414 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:35.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:35.414 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:35.078+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:35.415 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:35.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:35.415 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:35.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:35.415 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:35.077+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:35.415 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:35.078+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:35.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:35.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:35.412+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:35.413+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:36.366 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:36.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:36.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:36.056+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:36.057+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:36.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:36.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:36.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:36.077+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:36.368 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:36.078+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:36.364+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:36.364+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:36.364+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:36.365+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:37.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:37.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:37.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:37.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:37.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:37.050+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:37.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:37.050+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:37.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:37.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:37.330 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:37.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:37.330 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:37.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:37.330 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:37.106+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:37.589 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:37.565+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28788 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:37.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:37.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:37.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:37.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:37.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:37.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:37.589 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:37.327+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:38.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:38.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:38.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:38.008+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:38.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:38.009+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:38.331 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:38.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:38.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:38.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:38.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:38.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:38.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:38.107+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:38.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:38.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:38.328+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:38.329+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:39.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:39.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:39.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:39.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:39.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:39.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:39.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:39.059+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:39.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:39.022+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:39.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:39.023+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:39.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:39.023+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:39.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:39.024+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:39.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:39.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:39.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:39.330+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:40.305 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:40.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:40.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:40.060+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:40.061+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:40.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:40.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:40.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:40.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:40.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:40.035+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:40.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:40.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:40.303+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:40.304+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:41.289 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:41.038+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:41.038+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:41.038+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:41.039+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:41.290 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.009+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:41.291 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.009+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:41.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:41.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:41.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:41.288+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:42.026 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.973+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:42.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.973+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:42.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.973+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:42.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:41.973+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:42.313 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:42.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:42.313 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:42.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:42.313 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:42.024+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:42.313 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:42.025+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:42.574 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:42.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:42.574 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:42.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:42.574 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:42.311+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:42.574 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:42.311+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:42.846 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:42.569+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28881 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:43.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:43.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:43.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:43.073+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:43.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:43.074+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:43.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:42.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:43.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:42.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:43.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:42.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:43.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:42.985+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:43.634 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:43.352+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:43.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:43.352+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:43.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:43.352+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:43.635 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:43.353+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:44.319 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:44.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:44.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:44.093+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:44.094+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:43.976+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:43.976+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:44.320 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:43.976+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:44.321 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:43.977+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:44.317+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:44.317+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:44.317+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:44.318+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:45.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:45.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:45.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:45.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:45.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:45.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:45.364 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:45.118+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:45.365 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:44.999+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:45.365 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:44.999+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:45.365 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:44.999+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:45.365 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:44.999+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:45.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:45.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:45.362+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:45.363+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:46.124+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:46.124+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:46.124+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:46.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:46.125+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:46.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:45.954+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:46.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:45.954+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:46.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:45.954+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:46.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:45.955+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:46.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:46.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:46.330+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:46.330+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:47.321 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:47.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:47.321 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:47.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:47.321 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:47.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:47.321 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:47.143+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:47.322 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:47.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:47.322 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:47.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:47.322 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:47.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:47.322 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:47.004+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:47.575 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:47.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:47.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:47.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:47.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:47.318+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:47.576 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:47.320+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:47.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:47.572+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 28978 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:48.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:48.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:48.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:48.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:48.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:48.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:48.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:48.123+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:48.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:48.042+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:48.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:48.042+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:48.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:48.042+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:48.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:48.043+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:48.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:48.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:48.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:48.302+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:49.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:49.079+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:49.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:49.079+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:49.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:49.079+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:49.307 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:49.080+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:49.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:49.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:49.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:49.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:49.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:49.034+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:49.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:49.035+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:49.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:49.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:49.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:49.305+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:50.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:50.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:50.075+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:50.075+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:50.064+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:50.064+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:50.064+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:50.065+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:50.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:50.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:50.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:50.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:50.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:50.311+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:51.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:51.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:51.101+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:51.102+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:51.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:51.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:51.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:51.275+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:51.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:51.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:51.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:51.041+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:52.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:52.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:52.231+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:52.232+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:52.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:52.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:52.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:52.150+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:52.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:52.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:52.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:52.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:52.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:52.049+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:52.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:52.050+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:52.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:52.576+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29072 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:53.141+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:53.141+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:53.141+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:53.141+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:53.257+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:53.257+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:53.257+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:53.257+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:53.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:53.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:53.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:53.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:53.041+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:54.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:54.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:54.044+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:54.044+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:54.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:54.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:54.243+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:54.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:54.243+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:54.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:54.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:54.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:54.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:54.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:54.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:54.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:54.151+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:55.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:55.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:55.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:55.093+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:55.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:55.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:55.207+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:55.208+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:55.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:55.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:55.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:55.136+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:56.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:56.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:56.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:56.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:56.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:56.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:56.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:56.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:56.184+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:56.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:56.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:56.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:56.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:56.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:56.104+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:56.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:56.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:57.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:57.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:57.208+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:57.209+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:57.143+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:57.144+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:57.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:57.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:57.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:57.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:57.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:57.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:57.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:57.190+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:57.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:56:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:56:57.579+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29170 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:56:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:58.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:58.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:58.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:58.241+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:58.193+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:58.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:58.194+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:58.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:58.220+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:58.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:58.220+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:58.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:58.220+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:58.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:58.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:59.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:59.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:59.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:56:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:56:59.263+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:59.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:59.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:59.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:56:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:56:59.209+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:56:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:59.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:59.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:59.173+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:56:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:56:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:56:59.175+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:00.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:00.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:00.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:00.238+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:00.239+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:00.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:00.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:00.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:00.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:00.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:00.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:00.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:00.179+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:00.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:00.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:00.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:00.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:00.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:00.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:00.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:00.127+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:01.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:01.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:01.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:01.200+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:01.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:01.201+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:01.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:01.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:01.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:01.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:01.209+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:01.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:01.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:01.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:01.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:01.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:01.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:01.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:01.129+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:02.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:02.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:02.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:02.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:02.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:02.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:02.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:02.201+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:02.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:02.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:02.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:02.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:02.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:02.098+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:02.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:02.099+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:02.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:02.583+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29264 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:03.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:03.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:03.215+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:03.216+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:03.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:03.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:03.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:03.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:03.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:03.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:03.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:03.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:04.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:04.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:04.204+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:04.204+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:04.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:04.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:04.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:04.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:04.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:04.157+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:04.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:04.157+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:04.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:04.157+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:04.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:04.158+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:05.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:05.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:05.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:05.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:05.219+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:05.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:05.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:05.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:05.130+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:05.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:05.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:05.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:05.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:05.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:05.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:05.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:05.231+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:06.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:06.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:06.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:06.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:06.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:06.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:06.271+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:06.272+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:06.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:06.170+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:06.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:06.171+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:07.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:07.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:07.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:07.188+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:07.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:07.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:07.279+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:07.280+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:07.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:07.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:07.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:07.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:07.135+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:07.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:07.135+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:07.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:07.586+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29361 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:08.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:08.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:08.168+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:08.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:08.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:08.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:08.195+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:08.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:08.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:08.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:08.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:08.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:08.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:08.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:08.270+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:08.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:08.271+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:09.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:09.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:09.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:09.142+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:09.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:09.142+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:09.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:09.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:09.157+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:09.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:09.158+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:09.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:09.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:09.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:09.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:09.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:09.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:09.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:09.257+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:10.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:10.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:10.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:10.196+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:10.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:10.196+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:10.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:10.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:10.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:10.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:10.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:10.281+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:10.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:10.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:10.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:10.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:10.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:10.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:10.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:10.166+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:10.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:10.167+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:11.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:11.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:11.159+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:11.160+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:11.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:11.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:11.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:11.155+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:11.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:11.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:11.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:11.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:11.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:11.267+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:12.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:12.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:12.153+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:12.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:12.153+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:12.247+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:12.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:12.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:12.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:12.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:12.188+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:12.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:12.189+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:12.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:12.590+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29454 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:13.183+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:13.183+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:13.183+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:13.183+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:13.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:13.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:13.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:13.200+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:13.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:13.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:13.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:13.163+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:13.163+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:14.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:14.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:14.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:14.187+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:14.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:14.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:14.127+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:14.128+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:14.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:14.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:14.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:14.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:14.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:14.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:14.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:14.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:15.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:15.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:15.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:15.215+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:15.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:15.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:15.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:15.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:15.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:15.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:15.107+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:15.107+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:16.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:16.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:16.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:16.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:16.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:16.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:16.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:16.215+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:16.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:16.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:16.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:16.063+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:17.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:17.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:17.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:17.231+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:17.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:17.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:17.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:17.128+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:17.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:17.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:17.026+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:17.026+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:17.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:17.594+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29553 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:18.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:18.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:18.169+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:18.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:18.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:18.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:18.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:18.048+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:18.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:18.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:18.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:18.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:18.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:18.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:18.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:19.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:19.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:19.148+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:19.150+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:19.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:19.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:19.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:19.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:19.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:19.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:19.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:19.047+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:20.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:20.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:20.068+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:20.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:20.069+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:20.147+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:20.148+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:20.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:20.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:20.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:20.224+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:21.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:21.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:21.172+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:21.172+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:21.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:21.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:21.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:21.246+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:21.034+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:22.151+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:22.152+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:22.152+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:22.152+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:22.261+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:22.261+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:22.261+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:22.262+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:22.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:22.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:22.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:22.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:22.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:22.039+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:22.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:22.597+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29646 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:23.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:23.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:23.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:23.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:23.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:23.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:23.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:23.287+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:23.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:23.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:23.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:23.086+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:23.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:23.087+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:24.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:24.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:24.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:24.309+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:24.311+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:24.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:24.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:24.112+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:24.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:24.113+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:24.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:24.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:24.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:24.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:24.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:24.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:24.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:24.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:25.087+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:25.087+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:25.087+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:25.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:25.088+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:25.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:25.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:25.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:25.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:25.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:25.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:25.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:25.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:25.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:25.067+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:25.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:25.068+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:26.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:26.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:26.134 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:26.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:26.135 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:26.043+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:26.135 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:26.043+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:26.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:26.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:26.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:26.132+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:26.133+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:26.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:26.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:26.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:26.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:27.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:27.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:27.091+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:27.092+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:27.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:27.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:27.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:27.260+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:27.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:27.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:27.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:27.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:27.092+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:27.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:27.092+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:27.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:27.600+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29743 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:28.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:28.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:28.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:28.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:28.048+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:28.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:28.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:28.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:28.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:28.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:28.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:28.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:28.058+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:28.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:28.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:28.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:28.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:28.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:28.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:28.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:28.289+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:29.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:29.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:29.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:29.080+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:29.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:29.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:29.280+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:29.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:29.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:29.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:29.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:29.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:29.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:29.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:29.041+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:30.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:30.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:30.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:30.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:30.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:30.024+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:30.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:30.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:30.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:30.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:30.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:30.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:30.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:30.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:30.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:30.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:30.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:30.430 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:30.102+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:31.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:31.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:31.020+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:31.021+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:31.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:31.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:31.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:31.122+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:31.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:31.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:31.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:31.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:31.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:31.223+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:31.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:31.224+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:32.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:32.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:32.154+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:32.155+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:32.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:32.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:32.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:32.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:32.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:32.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:32.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:32.239+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:32.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:32.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:32.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:32.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.002+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:32.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:32.604+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29837 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:33.242 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:33.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:33.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:33.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:33.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:33.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:33.243 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:33.124+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:33.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:33.243 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:33.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.984+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:33.244 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:32.985+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:33.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:33.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:33.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:33.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:33.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:33.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:33.639 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:33.241+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:34.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:34.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:34.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:34.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:34.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:34.139+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:34.215 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:34.139+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:34.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:33.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:34.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:33.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:34.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:33.945+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:34.216 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:33.945+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:34.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:34.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:34.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:34.213+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:34.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:34.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:34.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:34.911+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:35.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:35.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:35.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:35.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:35.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:35.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:35.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:35.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:35.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:35.240+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:36.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:35.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:36.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:35.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:36.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:35.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:36.125 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:35.867+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:36.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:36.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:36.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:36.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:36.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:36.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:36.122+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:36.123+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:37.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:37.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:37.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:37.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:37.156 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:37.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:37.157 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:37.088+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:37.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:36.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:37.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:36.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:37.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:36.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:37.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:36.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:37.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:37.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:37.154+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:37.155+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:37.866 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:37.607+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 29936 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:38.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:37.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:38.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:37.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:38.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:37.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:38.140 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:37.865+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:38.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:38.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:38.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:38.184+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:38.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:38.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:38.137+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:38.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:39.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:38.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:39.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:38.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:39.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:38.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:39.092 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:38.822+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:39.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:39.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:39.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:39.199+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:39.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:39.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:39.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:39.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:40.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:40.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:40.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:40.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:40.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:40.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:40.173 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:40.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:39.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:39.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:39.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:40.174 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:39.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:40.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:40.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:40.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:40.172+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:41.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:40.819+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:41.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:40.819+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:41.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:40.819+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:41.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:40.820+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:41.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:41.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:41.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:41.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:41.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:41.123+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:41.123+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:41.123+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:41.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:41.124+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:42.050 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:41.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:42.050 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:41.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:42.051 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:41.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:42.051 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:41.852+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:42.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:42.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:42.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:42.118+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:42.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:42.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:42.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:42.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:42.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:42.048+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:42.429 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:42.049+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:42.876 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:42.611+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30030 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:43.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:43.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:43.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:43.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:43.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:43.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:43.146 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:43.028+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:43.147 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:42.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.147 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:42.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.147 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:42.874+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.147 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:42.875+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:43.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:43.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:43.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:43.145+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:43.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:43.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:43.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:43.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:44.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:44.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:44.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:44.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:44.066+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:44.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:44.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:44.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:44.154+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:45.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:45.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:45.087+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:45.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:45.087+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:44.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:44.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:44.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:45.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:44.888+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:45.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:45.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:45.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:45.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:45.192+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:46.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:46.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:46.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:46.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:46.169 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:46.079+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:46.170 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:46.079+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:46.170 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:45.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:46.170 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:45.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:46.171 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:45.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:46.171 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:45.849+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:46.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:46.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:46.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:46.167+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:47.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:46.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:46.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:46.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:46.800+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:47.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:47.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:47.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:47.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:47.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:47.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:47.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:47.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:47.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:47.149+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:47.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:47.614+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30127 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:47.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:47.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:47.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:47.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:47.800+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:48.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:48.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:48.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:48.072+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:48.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:48.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:48.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:48.153+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:49.103 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:48.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:49.103 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:48.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:49.103 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:48.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:49.103 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:48.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:49.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:49.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:49.102+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:49.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:49.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:49.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:49.175+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:50.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:50.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:50.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:50.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:50.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:50.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:50.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:49.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:50.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:49.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:50.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:49.849+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:50.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:49.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:50.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:50.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:50.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:50.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:50.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:50.200+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:51.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:50.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:51.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:50.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:51.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:50.845+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:51.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:50.846+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:51.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:51.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:51.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:51.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:51.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:51.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:51.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:51.175+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:52.111 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:51.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:51.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:51.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:51.862+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:52.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:52.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:52.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:52.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:52.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:52.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:52.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:52.159+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:52.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:52.618+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30221 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:52.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:52.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:52.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:52.865+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:53.146+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:53.146+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:53.146+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:53.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:53.147+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:53.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:53.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:53.095+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:53.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:53.096+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:54.116 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:53.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:54.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:53.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:54.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:53.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:54.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:53.861+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:54.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:54.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:54.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:54.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:54.186+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:54.186+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:54.186+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:54.187+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:55.144 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:55.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:55.144 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:55.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:55.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:55.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:55.145 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:55.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:55.146 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:54.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:55.146 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:54.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:55.146 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:54.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:55.146 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:54.852+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:55.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:55.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:55.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:55.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:55.142+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:56.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:55.832+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:56.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:55.832+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:56.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:55.832+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:56.112 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:55.833+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:56.109+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:56.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:56.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:56.110+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:56.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:56.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:56.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:56.186+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:57.113 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:56.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:57.113 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:56.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:57.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:56.851+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:57.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:56.851+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:57.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:57.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:57.111+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:57.112+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:57.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:57.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:57.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:57.173+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:57.889 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:57:57.622+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30320 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:57:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:58.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:58.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:58.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:58.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:58.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:57.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:57.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:57.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:57.888+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:58.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:58.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:58.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:58.212+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:59.101 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:58.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:59.102 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:58.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:59.102 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:58.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:57:59.102 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:58.843+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:59.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:59.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:59.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:57:59.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:57:59.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:57:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:59.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:59.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:59.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:57:59.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:57:59.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:59.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:59.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:59.854+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:57:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:57:59.854+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:00.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:00.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:00.057+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:00.057+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:00.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:00.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:00.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:00.238+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:00.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:00.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:00.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:00.888+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:01.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:01.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:01.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:01.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:01.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:01.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:01.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:01.196+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:02.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:01.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.075 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:01.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.075 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:01.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.075 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:01.856+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:02.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:02.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:02.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:02.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:02.189+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:02.189+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:02.189+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:02.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:02.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:02.626+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30413 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:02.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:02.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:02.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:02.851+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:03.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:03.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:03.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:03.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:03.028+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:03.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:03.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:03.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:03.231+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:03.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:03.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:03.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:03.861+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:04.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:04.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:04.024+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:04.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:04.024+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:04.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:04.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:04.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:04.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:04.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:04.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:04.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:04.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:04.868+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:05.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:05.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:05.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:05.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:05.065+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:05.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:05.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:05.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:05.233+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:06.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:05.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:06.110 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:05.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:06.110 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:05.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:06.110 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:05.826+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:06.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:06.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:06.107+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:06.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:06.108+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:06.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:06.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:06.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:06.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:07.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:06.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:06.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:06.806+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.091 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:06.808+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:07.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:07.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:07.088+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:07.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:07.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:07.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:07.179+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:07.179+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:07.179+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:07.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:07.180+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:07.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:07.629+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30510 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:07.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:07.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:07.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:07.853+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:08.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:08.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:08.105+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:08.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:08.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:08.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:08.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:08.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:08.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:08.199+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:09.101 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:08.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:09.101 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:08.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:09.101 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:08.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:09.101 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:08.890+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:09.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:09.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:09.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:09.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:09.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:09.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:09.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:09.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:09.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:09.225+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:10.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:10.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:10.138+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:10.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:10.138+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:09.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:10.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:09.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:10.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:09.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:10.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:09.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:10.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:10.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:10.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:10.215+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:11.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:10.898+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:11.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:10.898+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:11.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:10.898+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:11.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:10.899+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:11.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:11.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:11.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:11.154+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:11.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:11.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:11.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:11.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:12.131 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:11.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.131 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:11.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:11.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:11.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:12.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:12.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:12.446 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:12.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:12.446 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:12.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:12.446 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:12.230+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:12.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:12.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:12.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:12.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:12.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:12.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:12.447 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:12.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:12.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:12.633+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30604 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:12.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:12.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:12.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:12.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:12.808+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:13.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:13.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:13.641 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:13.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:13.174+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:13.175+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:13.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:13.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:13.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:13.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:13.185+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:13.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:13.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:13.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:13.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:13.802+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:14.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:14.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:14.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:14.161+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:14.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:14.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:14.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:14.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:14.186+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:15.148 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:14.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:15.148 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:14.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:15.148 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:14.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:15.148 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:14.795+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:15.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:15.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:15.145+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:15.146+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:15.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:15.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:15.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:15.218+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:16.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:16.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:16.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:16.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:15.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:16.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:16.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:16.155+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:16.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:16.156+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:16.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:16.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:16.238+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:16.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:16.239+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:17.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:16.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:16.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:16.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:16.759+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:17.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:17.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:17.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:17.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:17.191+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:17.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:17.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:17.141+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:17.142+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:17.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:17.637+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30703 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:17.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:17.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:17.754+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:17.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:17.754+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:18.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:18.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:18.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:18.213+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:18.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:18.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:18.136+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:18.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:18.137+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:19.135 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:18.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:19.136 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:18.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:19.136 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:18.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:19.136 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:18.746+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:19.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:19.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:19.133+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:19.134+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:19.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:19.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:19.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:46.768409+0000 (oldest deadline 2024-01-19T06:48:06.768409+0000) 2024-01-19T06:58:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:19.253+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:20.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:20.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:20.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:19.795+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:20.114 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:19.796+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:20.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:20.239+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:20.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:20.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:20.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:20.112+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:21.116 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:20.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:21.116 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:20.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:21.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:20.811+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:21.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:20.812+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:21.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:21.234+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:21.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:21.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:21.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:21.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:21.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:22.085 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:21.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.085 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:21.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:21.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:21.838+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:22.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:22.242+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:22.083+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:22.083+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:22.083+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:22.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:22.084+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:22.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:22.641+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30797 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:22.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:22.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:22.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:22.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:22.802+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:23.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:23.249+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:23.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:23.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:23.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:23.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:23.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:23.795+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:24.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:24.285+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:24.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:24.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:24.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:24.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:25.041 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:24.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:25.042 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:24.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:25.042 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:24.746+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:25.042 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:24.747+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:25.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:25.246+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:25.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:25.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:25.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:25.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:25.040+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:26.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:25.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:26.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:25.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:26.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:25.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:26.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:25.764+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:26.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:26.238+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:26.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:26.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:26.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:26.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:27.108 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:26.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:27.108 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:26.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:27.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:26.716+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:53.885595+0000 (oldest deadline 2024-01-19T06:48:13.885595+0000) 2024-01-19T06:58:27.109 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:26.717+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:27.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:27.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:27.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:27.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:27.106+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:27.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:27.645+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30894 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:27.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:27.682+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:28.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:28.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:28.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:28.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:28.094+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:28.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:28.095+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:29.128 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:28.684+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:29.261+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:29.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:29.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:29.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:29.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:29.127+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:29.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:29.672+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:30.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:30.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:30.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:30.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:30.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:30.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:30.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:30.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:30.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:30.084+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:30.640+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:31.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:31.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:31.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:31.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:31.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:31.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:31.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:31.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:31.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:31.596+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:32.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:32.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:32.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:32.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:32.170+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:32.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:32.171+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:32.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:32.648+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 30987 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:32.558+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:33.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:33.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:33.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:33.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:33.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:33.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:33.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:33.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:33.154+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:33.510+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:34.244+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:34.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:34.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:34.150+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:47:56.278101+0000 (oldest deadline 2024-01-19T06:48:16.278101+0000) 2024-01-19T06:58:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:34.151+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:34.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:34.497+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:35.112+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:35.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:35.281+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:35.465+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:36.078+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:36.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:36.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:36.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:36.437+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:37.082+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:37.316+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:37.779 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:37.653+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31086 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:37.779 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:37.456+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:38.132+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:38.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:38.284+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:38.697 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:38.433+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:39.424 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:39.161+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:39.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:39.287+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:39.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:39.422+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:40.114+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:40.282+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:40.427+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:41.399 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:41.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:41.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:41.262+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:41.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:41.397+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:42.379 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:42.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:42.379 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:42.280+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:42.654 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:42.377+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:42.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:42.656+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31180 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:43.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:43.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:43.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:43.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:43.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:43.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:43.366 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:43.262+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:43.367 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:43.263+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:43.642 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:43.364+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:44.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:44.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:44.412 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:44.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:44.412 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:44.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:44.412 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:44.286+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:44.412 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:44.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:44.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:44.409+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:45.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:45.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:45.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:45.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:45.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:45.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:45.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:45.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:45.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:45.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:45.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:45.402+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:46.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:46.075+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:46.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:46.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:46.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:46.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:46.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:46.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:46.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:46.203+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:46.675 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:46.399+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:47.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:47.106+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:47.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:47.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:47.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:47.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:47.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:47.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:47.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:47.247+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:47.662 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:47.371+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:47.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:47.661+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31275 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:48.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:48.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:48.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:48.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:48.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:48.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:48.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:48.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:48.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:48.219+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:48.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:48.403+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:49.410 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:49.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:49.410 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:49.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:49.411 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:49.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:49.411 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:49.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:49.411 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:49.173+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:49.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:49.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:49.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:49.409+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:49.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:49.410+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:50.371 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:50.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:50.372 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:50.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:50.372 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:50.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:50.372 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:50.166+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:50.372 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:50.166+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:50.370+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:50.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:50.371+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:51.091+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:51.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:51.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:51.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:51.140+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:51.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:51.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:51.330+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:51.331+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:52.116+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:52.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:52.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:52.318+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:52.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:52.319+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:52.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:52.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:52.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:52.190+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:52.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:52.664+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31369 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:53.343 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:53.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:53.344 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:53.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:53.344 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:53.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:53.344 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:53.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:53.345 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:53.215+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:53.643 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:53.341+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:53.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:53.341+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:53.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:53.341+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:53.644 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:53.342+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:54.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:54.065+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:54.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:54.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:54.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:54.293+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:54.294+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:54.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:54.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:54.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:54.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:54.231+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:55.266 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:55.053+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:55.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:55.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:55.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:55.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:55.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:55.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:55.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:55.304+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:55.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:55.305+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:56.326 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:56.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:56.326 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:56.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:56.327 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:56.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:56.327 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:56.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:56.327 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:56.290+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:56.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:56.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:56.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:56.324+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:56.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:56.325+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:57.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:57.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:57.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:57.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:57.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:57.062+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:57.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:57.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:57.339 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:57.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:57.340 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:57.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:57.340 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:57.308+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:57.340 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:57.309+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:57.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:57.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:57.671 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:57.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:57.672 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:57.337+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:57.672 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:57.337+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:57.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:58:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:58:57.668+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31468 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:58:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:58.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:58.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:58.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:58.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:58.315+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:58.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:58.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:58.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:58.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:58.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:58.098+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:58.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:58.099+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:58.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:58.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:58.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:58.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:58.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:58.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:58.429 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:58.302+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:59.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:59.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:59.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:58:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:58:59.314+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:59.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:59.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:59.113+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:58:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:58:59.114+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:58:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:59.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:59.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:59.342+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:58:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:58:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:58:59.343+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:00.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:00.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:00.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:00.337+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:00.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:00.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:00.126+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:00.127+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:00.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:00.328+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:00.328+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:01.357 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:01.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:01.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:01.315+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:01.358 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:01.315+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:01.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:01.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:01.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:01.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:01.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:01.134+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:01.359 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:01.135+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:01.355+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:01.355+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:01.355+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:01.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:01.356+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:02.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:02.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:02.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:02.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:02.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:02.268+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:02.356 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:02.269+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:02.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:02.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:02.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:02.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:02.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:02.099+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:02.357 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:02.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:02.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:02.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:02.353+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:02.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:02.354+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:02.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:02.672+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31562 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:03.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:03.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:03.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:03.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:03.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:03.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:03.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:03.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:03.317+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:03.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:03.318+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:03.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:03.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:03.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:03.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:03.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:03.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:03.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:03.404+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:04.338 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:04.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:04.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:04.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:04.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:04.082+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:04.339 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:04.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:04.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:04.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:04.377+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:04.377+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:04.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:04.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:04.336+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:04.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:04.337+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:05.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:05.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:05.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:05.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:05.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:05.038+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:05.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:05.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:05.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:05.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:05.364+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:05.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:05.365+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:05.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:05.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:05.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:05.371+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:06.335 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:06.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:06.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:06.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:06.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:06.076+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:06.336 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:06.077+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:06.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:06.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:06.399+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:06.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:06.400+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:06.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:06.333+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:06.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:06.333+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:06.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:06.333+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:06.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:06.334+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:07.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:07.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:07.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:07.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:07.104+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:07.105+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:07.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:07.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:07.336+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:07.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:07.336+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:07.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:07.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:07.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:07.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:07.383+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:08.060 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:07.677+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31658 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:08.377 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:08.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:08.377 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:08.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:08.377 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:08.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:08.377 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:08.059+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:08.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:08.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:08.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:08.376+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:08.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:08.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:08.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:08.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:08.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:08.387+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:08.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:08.387+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:09.393 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:09.045+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:09.393 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:09.045+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:09.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:09.045+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:09.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:09.046+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:09.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:09.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:09.403+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:09.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:09.403+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:09.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:09.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:09.391+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:09.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:09.392+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:10.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:10.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:10.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:10.068+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:10.384 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:10.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:10.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:10.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:10.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:10.382+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:10.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:10.383+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:10.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:10.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:10.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:10.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:10.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:10.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:10.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:10.393+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:11.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:11.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:11.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:11.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:11.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:11.042+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:11.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:11.042+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:11.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:11.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:11.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:11.426+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:11.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:11.427+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:11.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:11.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:11.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:11.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:11.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:11.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:11.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:11.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:12.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:12.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:12.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:12.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:12.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:12.092+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:12.387 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:12.092+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:12.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:12.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:12.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:12.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:12.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:12.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:12.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:12.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:12.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:12.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:12.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:12.475+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:12.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:12.475+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:13.121 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:12.681+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31753 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:13.374 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:13.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:13.374 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:13.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:13.374 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:13.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:13.374 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:13.120+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:13.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:13.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:13.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:13.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:13.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:13.407+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:13.645 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:13.408+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:13.646 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:13.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:13.646 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:13.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:13.646 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:13.473+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:13.646 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:13.473+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:14.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:14.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:14.123+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:14.123+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:14.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:14.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:14.492+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:14.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:14.493+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:14.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:14.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:14.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:14.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:14.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:14.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:14.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:14.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:15.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:15.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:15.132+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:15.133+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:15.518+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:15.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:15.519+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:15.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:15.519+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:15.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:15.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:15.481+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:15.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:15.482+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:16.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:16.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:16.115+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:16.117+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:16.501+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:16.501+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:16.501+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:16.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:16.501+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:16.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:16.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:16.458+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:16.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:16.459+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:17.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:17.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:17.097+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:17.098+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:17.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:17.684+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31851 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:17.708 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:17.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:17.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:17.453+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:17.454+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:17.546+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:17.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:17.547+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:17.709 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:17.547+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:18.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:18.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:18.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:18.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:18.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:18.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:18.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:18.581+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:18.581+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:18.473+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:18.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:18.473+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:19.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:19.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:19.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:19.102+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:19.467+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:19.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:19.468+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:19.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:19.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:19.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:19.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:19.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:20.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:20.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:20.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:20.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:20.121+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:20.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:20.121+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:20.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:20.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:20.502+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:20.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:20.503+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:20.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:20.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:20.631+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:20.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:20.632+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:21.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:21.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:21.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:21.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:21.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:21.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:21.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:21.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:21.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:21.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:21.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:21.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:21.672+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:22.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:22.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:22.129+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:22.130+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:22.811 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:22.689+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 31946 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:22.812 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:22.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:22.812 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:22.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:22.812 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:22.643+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:22.812 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:22.644+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:22.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:22.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:22.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:22.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:22.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:22.505+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:22.813 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:22.506+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:23.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:23.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:23.119+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:23.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:23.120+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:23.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:23.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:23.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:23.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:23.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:23.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:23.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:23.676+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:23.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:23.677+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:24.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:24.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:24.101+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:24.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:24.101+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:24.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:24.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:24.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:24.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:24.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:24.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:24.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:24.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:25.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:24.703+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:24.703+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:24.703+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:24.704+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:25.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:25.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:25.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:25.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:25.704 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:25.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:25.704 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:25.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:25.704 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:25.451+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:25.704 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:25.452+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:25.705 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:25.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.705 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:25.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.705 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:25.701+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:25.705 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:25.702+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:26.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:26.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:26.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:26.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:26.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:26.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:26.737 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:26.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:26.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:26.469+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:26.738 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:26.470+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:27.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:26.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:26.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:26.735+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:26.736+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:27.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:27.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:27.084+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:27.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:27.819 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:27.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:27.820 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:27.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:27.820 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:27.512+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:27.820 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:27.514+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:27.820 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:27.700+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.820 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:27.700+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:27.700+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:27.821 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:27.702+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:27.821 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:27.693+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32042 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:28.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:28.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:28.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:28.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:28.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:28.106+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:28.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:28.107+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:28.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:28.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:28.711+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:28.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:28.712+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:28.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:28.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:28.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:28.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:28.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:28.530+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:28.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:28.531+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:29.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:29.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:29.075+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:29.076+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:29.694 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:29.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:29.694 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:29.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:29.694 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:29.489+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:29.694 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:29.489+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:30.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:29.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:29.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:29.692+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:29.693+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:30.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:30.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:30.117+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:30.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:30.118+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:30.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:30.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:30.672+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:30.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:30.673+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:30.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:30.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:30.442+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:30.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:30.443+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:31.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:31.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:31.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:31.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:31.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:31.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:31.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:31.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:31.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:31.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:31.396+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:31.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:31.396+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:32.164 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:31.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.164 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:31.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:31.688+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:31.689+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:32.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:32.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:32.162+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:32.163+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:32.699 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:32.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:32.699 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:32.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:32.699 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:32.428+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:32.700 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:32.428+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:32.700 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:32.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.700 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:32.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.700 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:32.640+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:32.700 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:32.640+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:33.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:32.697+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32137 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:33.431 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:33.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:33.431 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:33.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:33.432 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:33.179+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:33.432 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:33.180+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:33.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:33.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:33.430+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:33.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:33.431+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:33.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:33.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:33.630+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:33.631+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:34.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:34.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:34.131+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:34.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:34.132+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:34.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:34.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:34.462+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:34.878 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:34.462+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:34.879 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:34.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:34.879 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:34.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:34.879 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:34.598+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:34.879 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:34.599+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:35.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:35.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:35.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:35.140+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:35.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:35.141+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:35.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:35.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:35.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:35.475+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:35.927 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:35.476+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:35.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:35.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:35.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:35.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:35.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:35.611+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:35.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:35.612+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:36.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:36.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:36.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:36.112+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:36.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:36.113+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:36.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:36.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:36.437+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:36.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:36.438+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:36.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:36.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:36.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:36.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:36.660+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:37.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:37.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:37.096+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:37.097+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:37.724 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:37.701+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32235 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:37.724 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:37.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:37.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:37.459+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:37.460+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:37.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:37.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:37.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:37.725 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:37.651+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:38.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:38.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:38.060+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:38.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:38.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:38.432+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:38.697 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:38.433+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:38.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:38.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:38.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:38.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:38.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:38.618+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:38.698 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:38.619+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:39.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:39.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:39.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:39.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:39.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:39.063+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:39.405 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:39.063+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:39.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:39.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:39.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:39.402+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:39.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:39.403+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:39.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:39.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:39.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:39.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:39.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:39.615+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:39.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:39.616+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:40.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:40.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:40.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:40.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:40.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:40.093+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:40.358 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:40.094+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:40.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:40.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:40.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:40.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:40.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:40.356+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:40.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:40.357+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:40.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:40.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:40.650+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:40.650+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:41.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:41.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:41.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:41.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:41.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:41.073+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:41.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:41.074+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:41.651 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:41.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:41.651 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:41.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:41.651 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:41.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:41.651 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:41.365+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:41.648+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:41.648+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:41.648+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:41.649+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:42.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:42.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:42.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:42.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:42.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:42.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:42.407 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:42.070+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:42.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:42.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:42.404+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:42.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:42.406+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:42.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:42.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:42.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:42.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:42.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:42.614+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:42.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:42.615+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:43.074 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:42.705+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32330 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:43.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:43.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:43.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:43.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:43.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:43.072+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:43.362 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:43.073+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:43.648 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:43.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:43.648 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:43.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:43.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:43.360+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:43.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:43.360+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:43.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:43.654+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:43.655+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:44.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:44.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:44.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:44.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:44.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:44.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:44.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:44.071+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:44.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:44.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:44.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:44.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:44.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:44.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:44.662 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:44.383+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:44.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:44.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:44.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:44.659+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:44.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:44.660+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:45.385 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:45.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:45.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:45.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:45.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:45.027+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:45.386 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:45.028+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:45.660 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:45.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:45.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:45.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:45.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:45.383+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:45.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:45.384+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:45.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:45.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:45.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:45.658+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:45.659+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:46.400 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:46.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:46.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:46.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:46.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:46.053+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:46.401 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:46.054+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:46.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:46.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:46.398+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:46.399+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:47.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:46.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:46.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:46.695+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:46.696+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:47.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:47.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:47.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:47.380 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.022+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:47.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:47.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:47.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:47.379+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:47.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:47.710+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32426 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:47.996 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:47.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.996 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:47.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.996 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:47.704+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:47.996 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:47.705+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:48.332 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:48.333 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:48.333 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.994+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:48.333 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:47.995+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:48.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:48.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:48.330+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:48.331+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:49.032 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:48.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.032 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:48.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.032 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:48.708+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.032 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:48.709+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:49.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:49.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:49.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:49.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:49.299 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:49.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:49.300 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:49.031+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:49.673 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:49.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:49.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:49.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:49.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:49.297+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:49.674 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:49.298+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:49.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:49.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:49.671+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:49.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:49.672+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:50.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:50.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:50.039+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:50.039+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:50.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:50.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:51.084 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:50.709+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:51.084 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:50.709+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:51.085 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:50.709+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:51.085 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:50.709+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:51.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:51.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:51.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:51.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:51.083+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:51.323+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:51.323+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:51.323+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:51.324+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:52.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:51.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:52.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:51.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:52.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:51.697+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:52.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:51.698+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:52.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:52.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:52.116+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:52.117+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:52.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:52.319+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:52.319+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:52.319+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:52.320+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:53.092 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:52.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.092 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:52.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.093 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:52.734+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.093 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:52.735+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:53.093 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:52.714+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32521 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:53.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:53.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:53.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:53.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:53.366 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:53.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:53.367 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:53.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:53.649 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:53.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:53.650 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:53.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:53.650 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:53.364+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:53.650 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:53.365+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:53.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:53.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:53.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:53.768+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:54.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:54.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:54.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:54.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:54.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:54.102+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:54.395 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:54.103+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:54.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:54.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:54.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:54.393+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:55.088 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:54.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:55.088 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:54.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:55.088 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:54.767+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:55.088 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:54.768+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:55.369 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:55.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:55.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:55.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:55.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:55.086+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:55.370 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:55.087+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:55.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:55.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:55.367+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:55.368+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:56.046 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:55.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:56.046 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:55.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:56.046 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:55.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:56.046 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:55.791+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:56.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:56.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:56.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:56.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:56.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:56.044+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:56.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:56.045+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:56.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:56.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:56.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:56.410+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:56.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:56.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:56.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:57.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:56.763+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:57.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:57.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:57.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:57.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:57.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:57.006+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:57.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:57.007+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:57.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:57.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:57.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:57.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:57.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:57.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:57.720 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:57.437+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:58.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:57.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:58.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:57.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:58.024 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:57.730+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:58.025 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:57.731+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:58.025 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 06:59:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T06:59:57.717+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32619 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T06:59:58.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:58.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:58.403 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:58.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:58.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:58.022+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:58.404 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:58.023+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:58.401+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:58.401+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:58.401+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:58.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:58.402+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:59.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:58.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:59.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:58.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:59.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:58.744+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T06:59:59.007 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:58.745+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:59.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:59.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:59.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.005+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T06:59:59.394 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.005+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T06:59:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:59.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:59.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:59.392+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T06:59:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T06:59:59.392+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:59.765+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:59.765+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:59.765+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T06:59:59.766+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.986+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:00.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 06:59:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T06:59:59.987+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:00.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:00.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:00.436+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:00.437+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:00.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:00.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:00.796+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:00.797+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:00.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:00.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:00.998+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:01.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:00.999+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:01.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:01.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:01.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:01.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:01.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:01.484+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:01.753 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:01.485+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:02.018 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:01.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:02.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:01.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:02.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:01.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:02.019 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:01.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:02.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:02.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:02.016+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:02.017+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:02.848 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:02.722+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32714 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:02.848 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:02.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:02.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:02.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:02.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:02.507+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:02.849 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:02.508+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:02.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:02.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:03.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:02.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:03.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:02.867+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:03.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:03.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:03.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:03.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:03.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:03.047+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:03.178 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:03.048+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:03.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:03.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:03.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:03.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:03.859 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:03.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:03.860 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:03.548+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:04.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:04.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:04.018+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:04.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:04.019+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:04.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:03.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:04.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:03.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:04.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:03.857+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:04.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:03.858+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:04.855 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:04.597+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:04.855 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:04.597+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:04.855 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:04.597+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:04.855 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:04.598+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:05.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:05.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:05.002+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:05.002+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:04.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:04.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:04.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:04.854+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:05.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:05.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:05.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:05.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:05.836 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:05.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:05.837 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:05.570+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:06.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:05.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:05.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:05.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:05.835+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:06.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:06.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:06.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:06.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:06.050+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:06.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:06.051+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:06.863 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:06.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:06.863 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:06.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:06.863 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:06.573+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:06.864 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:06.573+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:07.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:07.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:07.059+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:07.060+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:06.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:06.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:06.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:06.862+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:07.840 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:07.726+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32810 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:07.840 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:07.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:07.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:07.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:07.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:07.557+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:07.841 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:07.558+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:08.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:08.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:08.030+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:08.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:08.030+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:08.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:07.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:08.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:07.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:08.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:07.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:08.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:07.839+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:08.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:08.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:08.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:08.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:08.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:08.598+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:08.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:08.599+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:09.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:09.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:09.058+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:09.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:09.059+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:09.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:08.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:09.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:08.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:09.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:08.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:09.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:08.872+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:09.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:09.568+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:09.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:09.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:09.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:09.569+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:09.917 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:09.569+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:09.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:09.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:09.915+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:09.916+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:10.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:10.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:10.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:10.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:10.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:10.100+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:10.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:10.100+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:10.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:10.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:10.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:10.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:10.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:10.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:10.888 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:10.526+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:11.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:11.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:11.078+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:11.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:11.079+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:10.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:10.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:10.886+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:11.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:10.886+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:11.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:11.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:11.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:11.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:11.524+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:12.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:12.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:12.089+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:12.090+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:12.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:11.932+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:12.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:11.932+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:12.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:11.932+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:12.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:11.932+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:12.855 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:12.730+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 32905 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:12.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:12.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:12.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:12.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:12.856 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:12.520+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:13.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:13.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:13.046+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:13.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:13.047+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:12.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:12.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:12.961+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:13.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:12.962+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:13.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:13.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:13.497+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:13.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:13.498+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:14.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:14.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:14.033+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:14.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:14.034+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:14.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:13.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:13.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:13.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:14.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:13.964+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:14.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:14.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:14.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:14.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:14.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:14.493+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:14.814 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:14.494+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:15.073 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:14.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:15.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:14.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:15.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:14.921+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:15.074 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:14.922+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:15.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:15.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:15.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:15.071+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:15.072+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:15.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:15.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:15.518+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:15.925 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:15.519+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:16.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:15.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:16.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:15.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:16.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:15.923+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:16.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:15.924+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:16.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:16.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:16.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:16.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:16.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:16.069+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:16.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:16.069+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:16.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:16.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:16.523+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:16.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:16.523+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:16.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:16.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:16.966+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:16.966+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:17.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:17.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:17.065+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:17.066+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:17.859 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:17.734+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33003 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:17.860 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:17.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:17.860 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:17.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:17.860 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:17.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:17.860 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:17.525+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:18.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:18.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:18.037+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:18.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:18.038+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:17.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:17.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:17.946+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:18.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:17.946+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:18.909 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:18.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:18.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:18.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:18.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:18.553+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:18.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:18.554+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:19.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:19.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:19.081+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:19.082+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:18.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:18.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:18.908+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:18.909+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:19.918 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:19.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:19.919 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:19.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:19.919 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:19.532+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:19.919 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:19.533+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:20.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:20.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:20.110+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:20.111+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:19.917+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:20.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:19.917+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:20.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:19.917+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:20.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:19.918+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:20.872 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:20.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:20.872 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:20.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:20.872 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:20.556+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:20.873 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:20.557+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:21.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:21.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:21.090+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:21.091+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:20.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:20.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:20.871+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:21.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:20.872+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:21.892 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:21.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:21.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:21.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:21.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:21.525+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:21.893 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:21.526+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:21.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:21.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:21.890+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:21.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:22.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:22.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:22.085+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:22.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:22.085+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:22.865 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:22.738+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33095 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:22.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:22.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:22.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:22.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:22.865 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:22.520+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:22.866 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:22.521+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:23.116 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:22.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:23.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:22.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:23.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:22.916+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:23.117 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:22.917+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:23.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:23.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:23.114+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:23.115+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:23.909 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:23.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:23.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:23.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:23.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:23.515+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:23.910 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:23.515+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:24.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:24.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:24.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:24.128+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:24.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:24.129+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:24.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:23.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:24.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:23.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:24.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:23.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:24.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:23.908+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:24.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:24.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:24.891 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:24.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:24.891 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:24.547+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:24.891 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:24.547+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:25.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:24.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:25.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:24.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:25.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:24.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:25.156 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:24.889+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:25.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:25.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:25.153+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:25.155+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:25.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:25.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:25.535+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:25.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:25.535+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:26.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:26.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:26.160+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:26.161+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:25.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:25.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:25.928+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:25.929+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:26.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:26.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:26.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:26.499+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:26.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:26.500+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:26.891 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:26.500+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:26.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:26.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:26.888+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:26.889+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:27.467 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:27.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:27.467 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:27.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:27.467 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:27.176+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:27.467 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:27.177+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:27.744 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:27.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:27.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:27.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:27.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:27.465+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:27.745 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:27.466+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:28.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:27.742+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33185 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:27.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:27.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:27.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:27.869+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:28.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:28.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:28.182+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:28.182+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:28.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:28.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:28.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:28.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:28.418+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:28.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:28.419+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:28.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:28.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:28.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:28.868+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:29.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:29.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:29.216+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:29.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:29.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:29.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:29.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:29.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:29.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:29.399+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:29.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:29.399+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:29.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:29.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:29.864+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:29.865+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:30.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:30.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:30.217+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:30.217+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:30.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:30.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:30.358+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:30.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:30.359+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:30.903+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:30.903+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:30.903+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:30.904+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:31.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:31.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:31.261+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:31.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:31.261+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:31.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:31.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:31.371+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:31.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:31.372+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:31.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:31.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:31.872+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:31.872+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:32.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:32.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:32.218+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:32.218+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:32.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:32.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:32.378+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:32.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:32.378+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:33.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:32.747+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33274 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:32.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:32.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:32.922+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:32.923+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:33.652 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:33.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:33.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:33.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:33.376+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:33.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:33.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:33.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:33.653 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:33.265+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:34.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:33.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:34.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:33.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:34.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:33.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:34.251 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:33.958+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:34.328+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:34.328+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:34.328+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:34.329+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:34.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:34.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:34.249+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:34.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:34.250+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:35.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:34.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:35.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:34.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:35.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:34.987+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:35.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:34.988+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:35.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:35.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:35.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:35.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:35.260+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:35.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:35.260+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:35.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:35.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:35.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:35.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:35.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:36.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:36.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:36.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:36.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:36.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:36.035+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:36.306 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:36.036+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:36.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:36.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:36.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:36.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:36.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:36.237+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:36.307 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:36.238+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:36.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:36.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:36.304+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:36.304+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:37.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:37.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:37.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:37.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:37.282+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:37.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:37.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:37.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:37.265+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:37.266+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:37.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:37.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:37.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:37.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:37.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:37.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:38.113 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:37.751+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33363 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:38.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:38.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:38.111+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:38.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:38.112+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:38.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:38.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:38.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:38.220+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:38.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:38.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:38.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:38.327+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:38.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:38.328+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:39.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:39.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:39.344+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:39.344+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:39.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:39.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:39.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:39.268+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:39.268+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:39.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:39.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:39.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:39.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:39.114+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:40.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:40.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:40.232+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:40.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:40.232+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:40.109+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:40.109+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:40.109+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:40.110+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:40.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:40.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:40.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:40.332+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:40.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:40.334+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:41.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:41.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:41.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:41.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:41.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:41.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:41.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:41.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:41.127+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:41.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:41.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:41.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:41.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:41.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:41.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:41.240+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:42.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:42.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:42.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:42.341+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:42.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:42.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:42.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:42.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:42.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:42.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:42.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:42.119+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:43.122 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:42.755+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33449 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:43.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:43.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:43.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:43.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:43.338+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:43.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:43.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:43.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:43.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:43.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:43.255+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:43.255+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:43.255+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:43.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:43.256+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:44.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:44.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:44.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:44.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:44.082+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:44.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:44.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:44.296+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:44.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:44.297+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:44.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:44.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:44.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:44.301+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:44.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:44.301+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:45.306 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:45.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:45.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:45.256+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:45.257+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:45.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:45.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:45.053+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:45.307 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:45.054+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:45.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:45.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:45.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:45.305+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:46.310 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:46.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:46.310 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:46.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:46.310 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:46.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:46.310 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:46.230+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:46.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:46.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:46.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:46.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:46.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:46.055+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:46.311 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:46.056+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:46.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:46.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:46.308+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:46.309+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:47.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:47.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:47.095+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:47.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:47.096+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:47.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:47.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:47.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:47.204+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:47.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:47.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:47.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:47.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:47.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:47.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:47.317+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:48.062 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:47.759+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33539 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:48.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:48.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:48.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:48.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:48.198+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:48.278+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:48.278+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:48.278+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:48.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:48.279+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:48.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:48.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:48.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:48.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:48.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:48.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:48.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:48.061+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:49.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:49.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:49.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:49.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:49.209+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:49.210+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:49.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:49.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:49.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:49.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:49.045+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:49.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:49.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:49.301+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:49.302+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:50.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:50.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:50.083+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:50.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:50.084+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:50.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:50.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:50.274+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:50.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:50.275+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:50.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:50.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:50.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:50.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:50.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:50.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:50.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:50.246+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:51.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:51.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:51.277+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:51.277+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:51.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:51.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:51.250+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:51.251+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:51.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:51.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:51.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:51.096+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:51.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:51.097+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:52.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:52.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:52.059+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:52.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:52.060+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:52.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:52.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:52.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:52.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:52.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:52.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:52.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:52.248+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:52.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:52.249+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:53.081 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:52.764+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33631 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:53.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:53.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:53.267+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:53.267+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:53.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:53.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:53.079+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:53.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:53.080+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:53.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:53.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:53.266+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:53.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:53.267+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:54.314+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:54.314+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:54.314+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:54.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:54.315+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:54.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:54.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:54.285+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:54.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:54.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:54.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:54.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:54.128+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:54.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:54.129+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:55.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:55.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:55.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:55.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:55.106+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:55.291+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:55.291+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:55.291+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:55.291+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:55.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:55.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:55.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:55.292+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:56.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:56.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:56.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:56.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:56.141+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:56.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:56.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:56.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:56.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:56.285+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:56.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:56.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:56.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:56.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:56.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:56.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:56.314+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:57.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:57.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:57.281+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:57.282+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:57.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:57.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:57.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:57.299+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:57.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:57.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:57.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:57.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:57.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:57.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:57.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:57.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:58.152 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:00:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:00:57.768+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33729 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:00:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:58.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:58.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:58.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:58.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:58.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:58.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:58.304+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:58.305+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:58.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:58.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:58.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:58.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:58.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:58.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:58.151+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:59.234+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:59.234+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:59.234+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:00:59.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:00:59.235+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:59.325+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:00:59.326+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:59.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:59.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:59.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:59.133+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:00:59.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:00:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:00:59.134+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:00.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:00.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:00.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:00.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:00.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:00.237+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:00.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:00.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:00.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:00.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:00.329+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:00.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:00.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:00.106+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:00.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:00.107+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:01.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:01.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:01.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:01.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:01.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:01.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:01.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:01.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:01.314+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:01.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:01.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:01.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:01.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:01.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:01.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:01.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:01.250+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:02.149+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:02.149+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:02.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:02.149+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:02.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:02.149+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:02.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:02.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:02.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:02.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:02.211+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:02.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:02.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:02.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:02.271+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:02.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:02.271+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:02.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:02.271+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:03.179 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:02.772+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33824 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:03.653 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:03.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:03.653 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:03.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:03.653 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:03.195+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:03.653 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:03.197+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:03.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:03.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:03.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:03.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:03.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:03.272+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:03.654 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:03.272+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:03.654 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:03.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:03.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:03.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:03.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:03.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:03.655 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:03.253+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:04.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:04.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:04.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:04.260+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:04.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:04.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:04.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:04.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:04.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:04.220+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:04.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:04.221+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:04.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:04.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:04.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:04.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:04.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:04.283+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:04.679 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:04.284+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:05.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:05.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:05.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:05.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:05.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:05.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:05.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:05.287+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:05.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:05.288+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:05.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:05.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:05.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:05.174+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:05.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:05.174+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:06.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:06.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:06.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:06.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:06.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:06.256+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:06.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:06.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:06.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:06.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:06.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:06.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:06.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:06.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:06.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:06.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:06.150+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:06.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:06.151+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:07.229+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:07.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:07.229+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:07.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:07.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:07.305+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:07.305+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:07.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:07.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:07.190+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:07.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:07.191+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:08.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:07.777+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 33920 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:08.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:08.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:08.278+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:08.279+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:08.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:08.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:08.228+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:08.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:08.229+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:08.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:08.181+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:08.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:08.181+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:08.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:08.181+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:08.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:08.182+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:09.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:09.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:09.201+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:09.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:09.202+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:09.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:09.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:09.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:09.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:09.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:09.253+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:09.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:09.253+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:09.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:09.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:09.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:09.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:09.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:09.208+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:09.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:09.209+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:10.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:10.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:10.243+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:10.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:10.243+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:10.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:10.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:10.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:10.180+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:10.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:10.180+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:10.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:10.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:10.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:10.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:10.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:10.175+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:10.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:10.176+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:11.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:11.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:11.220+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:11.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:11.221+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:11.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:11.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:11.287+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:11.288+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:11.168+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:11.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:11.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:11.169+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:11.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:11.169+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:12.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:12.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:12.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:12.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:12.267+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:12.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:12.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:12.247+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:12.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:12.120+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:12.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:12.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:12.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:12.121+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:12.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:12.121+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:13.149 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:12.781+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34015 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:13.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:13.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:13.206+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:13.207+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:13.247+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:13.247+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:13.247+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:13.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:13.248+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:13.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:13.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:13.147+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:13.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:13.148+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:14.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:14.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:14.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:14.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:14.265+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:14.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:14.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:14.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:14.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:14.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:14.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:14.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:14.213+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:14.677 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:14.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:14.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:14.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:14.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:14.196+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:14.678 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:14.197+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:15.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:15.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:15.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:15.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:15.255+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:15.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:15.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:15.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:15.210+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:15.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:15.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:15.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:15.154+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:15.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:15.154+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:16.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:16.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:16.279+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:16.280+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:16.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:16.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:16.214+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:16.214+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:16.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:16.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:16.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:16.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:16.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:17.138+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:17.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:17.139+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:17.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:17.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:17.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:17.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:17.296+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:17.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:17.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:17.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:17.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:17.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:17.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:17.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:17.219+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:18.105 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:17.786+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34113 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:18.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:18.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:18.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:18.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:18.310+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:18.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:18.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:18.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:18.241+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:18.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:18.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:18.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:18.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:18.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:18.104+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:19.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:19.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:19.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:19.309+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:19.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:19.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:19.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:19.221+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:19.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:19.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:19.119+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:19.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:19.119+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:20.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:20.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:20.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:20.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:20.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:20.289+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:20.177+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:20.177+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:20.177+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:20.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:20.177+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:20.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:20.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:20.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:20.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:20.430 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:20.127+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:21.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:21.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:21.251+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:21.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:21.252+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:21.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:21.160+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:21.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:21.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:21.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:21.113+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:21.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:21.114+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:22.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:22.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:22.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:22.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:22.398 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:22.296+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:22.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:22.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:22.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:22.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:22.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:22.174+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:22.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:22.175+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:22.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:22.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:22.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:22.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:22.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:22.066+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:22.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:22.067+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:23.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:22.790+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34208 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.025+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:23.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.026+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:23.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:23.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:23.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:23.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:23.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:23.264+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:23.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:23.265+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:23.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:23.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:23.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:23.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:23.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:23.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:23.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:23.213+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:24.248 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:24.248 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:24.248 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.979+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:24.248 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:23.980+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:24.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:24.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:24.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:24.246+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:24.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:24.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:24.295+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:24.296+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:25.291 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:25.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:25.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:25.027+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:25.028+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:25.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:25.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:25.292 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:25.232+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:25.293 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:25.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:25.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:25.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:25.289+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:25.290+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:26.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:26.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:26.333 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:26.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:26.235+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:26.236+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:26.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:26.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:26.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:26.334 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:26.077+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:26.331+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:26.331+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:26.331+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:26.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:26.332+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:27.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:27.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:27.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:27.299+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:27.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:27.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:27.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:27.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:27.108+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:27.108+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:27.108+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:27.109+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:28.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:27.794+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34304 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:28.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:28.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:28.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:28.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:28.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:28.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:28.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:28.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:28.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:28.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:28.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:28.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:28.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:28.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:28.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:28.130+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:28.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:28.131+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:29.425 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:29.354+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:29.354+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:29.354+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:29.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:29.355+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:29.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:29.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:29.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:29.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:29.185+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:29.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:29.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:29.140+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:29.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:29.141+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:30.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:30.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:30.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:30.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:30.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:30.333+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:30.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:30.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:30.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:30.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:30.209+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:30.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:30.114+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:30.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:30.114+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:30.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:30.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:30.429 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:30.115+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:31.331 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:31.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:31.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:31.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:31.230+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:31.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:31.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:31.332 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:31.072+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:31.333 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:31.073+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:31.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:31.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:31.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:31.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:31.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:31.329+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:31.614 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:31.330+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:32.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:32.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:32.185+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:32.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:32.186+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:32.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:32.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:32.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:32.284+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:32.284+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:32.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:32.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:32.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:32.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:32.074+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:33.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:32.799+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34399 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:33.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:33.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:33.040+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:33.040+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:33.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:33.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:33.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:33.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:33.655 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:33.309+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:33.656 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:33.310+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:33.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:33.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:33.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:33.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:33.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:33.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:33.656 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:33.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:34.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:34.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:34.263+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:34.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:34.264+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:34.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:34.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:34.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:34.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:34.203+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:34.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:34.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:34.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:34.089+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:34.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:34.090+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:35.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:35.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:35.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:35.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:35.046+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:35.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:35.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:35.178+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:35.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:35.179+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:35.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:35.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:35.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:35.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:35.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:35.292+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:35.428 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:35.292+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:36.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:36.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:36.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:36.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:36.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:36.203+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:36.334 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:36.204+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:36.334 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:36.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:36.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:36.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:36.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:36.076+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:36.335 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:36.076+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:36.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:36.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:36.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:36.332+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:36.332+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:37.375 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:37.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:37.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:37.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:37.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:37.240+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:37.376 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:37.241+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:37.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:37.376 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:37.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:37.377 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.037+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:37.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:37.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:37.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:37.373+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:37.375+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:38.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:37.803+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34497 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.998+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:37.999+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:38.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:38.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:38.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:38.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:38.194+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:38.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:38.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:38.421+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:38.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:38.422+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:39.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:39.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:39.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:39.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:39.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:39.194+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:38.990+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:38.990+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:38.990+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:39.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:38.991+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:39.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:39.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:39.451+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:39.452+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.030+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:40.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.030+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:40.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:40.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:40.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:40.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:40.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:40.219+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:40.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:40.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:40.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:40.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:40.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:40.427+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:40.830 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:40.427+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.997+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.997+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.997+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:41.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:40.998+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:41.517 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:41.456+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:41.517 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:41.456+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:41.456+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:41.458+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:41.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:41.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:41.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:41.518 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:41.217+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:42.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:42.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:42.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:42.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:42.252+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:42.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:42.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:42.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:42.013+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:42.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:42.014+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:42.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:42.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:42.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:42.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:42.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:42.441+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:42.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:42.442+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:43.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:42.808+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34592 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:43.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:43.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:43.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.001+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:43.178 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.002+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:43.467 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:43.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:43.467 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:43.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:43.467 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:43.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:43.467 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:43.208+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:43.902 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:43.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:43.902 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:43.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:43.902 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:43.465+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:43.902 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:43.465+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:44.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:44.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:44.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.963+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:44.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:43.964+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:44.509 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:44.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:44.509 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:44.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:44.509 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:44.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:44.509 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:44.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:44.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:44.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:44.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:44.507+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:44.508+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:45.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:45.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:45.230+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:45.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:45.230+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:45.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:44.988+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:44.988+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:44.988+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:45.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:44.989+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:45.539+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:45.539+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:45.539+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:45.540+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:46.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:46.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:46.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:46.236+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:46.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:46.236+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:46.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:46.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:46.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:46.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:46.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:46.015+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:46.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:46.016+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:46.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:46.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:46.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:46.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:46.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:46.537+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:46.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:46.538+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:47.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:47.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:47.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:47.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:47.197+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:47.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:47.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:47.063+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:47.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:47.063+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:47.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:47.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:47.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:47.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:47.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:47.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:47.816 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:47.511+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:48.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:47.812+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34688 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:48.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:48.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:48.051+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:48.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:48.051+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:48.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:48.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:48.464+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:48.465+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:48.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:48.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:48.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:48.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:48.224+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:49.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:49.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:49.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:49.259+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:49.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:49.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:49.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:49.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:49.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:49.039+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:49.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:49.040+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:49.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:49.460+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:49.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:49.460+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:49.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:49.460+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:49.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:49.461+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.003+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:50.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.005+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:50.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:50.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:50.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:50.212+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:50.213+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:50.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:50.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:50.480+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:50.481+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.994+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:51.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:50.995+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:51.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:51.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:51.239+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:51.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:51.240+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:51.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:51.448+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:51.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:51.448+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:51.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:51.448+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:51.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:51.448+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:52.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:51.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:51.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:51.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:51.965+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:52.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:52.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:52.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:52.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:52.420+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:52.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:52.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:52.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:52.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:52.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:52.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:52.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:52.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:53.118 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:52.817+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34784 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:53.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:52.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:53.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:52.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:53.118 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:52.936+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:53.119 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:52.937+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:53.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:53.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:53.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:53.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:53.423 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:53.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:53.423 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:53.201+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:53.420+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:53.420+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:53.420+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:53.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:53.421+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:53.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:53.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:53.905+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:54.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:53.906+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:54.364 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-01-19T07:01:54.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-19T07:01:54.370 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-01-19T07:01:54.371 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-19T07:01:54.386 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-19T07:01:54.397 INFO:tasks.cephadm:Teardown begin 2024-01-19T07:01:54.397 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-19T07:01:54.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-19T07:01:54.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:01:54.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:01:54.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:54.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:54.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:54.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:54.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:54.195+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:54.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:54.196+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:54.893 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:54.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:54.893 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:54.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:54.893 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:54.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:54.894 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:54.455+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:55.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:54.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:54.892+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:55.422+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:55.422+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:55.422+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:55.422+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:55.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:55.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:55.210+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:55.211+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:56.190 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:55.935+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:56.190 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:55.935+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:56.190 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:55.935+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:56.190 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:55.936+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:56.449 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:56.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:56.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:56.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:56.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:56.246+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:56.450 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:56.247+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:56.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:56.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:56.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:56.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:56.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:56.447+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:56.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:56.449+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:57.291 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:56.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:57.291 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:56.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:57.291 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:56.951+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:57.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:56.951+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:57.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:01:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:57.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:57.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:57.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:57.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:57.474+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:57.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:57.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:57.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:57.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:57.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:57.289+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:57.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:57.290+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:58.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:01:57.821+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34881 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:01:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:57.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:57.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:57.958+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:58.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:57.959+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:58.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:58.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:58.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:58.481+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:58.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:58.482+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:58.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:58.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:58.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:58.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:58.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:58.300+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:58.678 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:58.300+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:59.261 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:59.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:59.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:01:59.262 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.007+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:59.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:59.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:59.259+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:01:59.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:01:59.261+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:01:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:59.473+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:59.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:59.473+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:59.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:59.473+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:01:59.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:01:59.474+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:00.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.957+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:00.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.957+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:00.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.957+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:00.233 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:01:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:01:59.958+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:00.498 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:00.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:00.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:00.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:00.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:00.258+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:00.499 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:00.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:00.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:00.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:00.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:00.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:00.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:00.496+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:00.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:00.497+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:01.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:01.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:01.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:01.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:01.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:01.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:01.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:01.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:01.006+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:01.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:01.006+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:01.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:01.455+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:01.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:01.455+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:01.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:01.455+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:01.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:01.455+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:02.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:02.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:02.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:02.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:02.335 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:02.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:02.336 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:02.225+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:02.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:02.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:02.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:02.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:02.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:02.046+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:02.336 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:02.047+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:02.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:02.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:02.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:02.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:02.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:02.434+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:03.080 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:02.825+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 34977 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:03.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:03.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:03.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:03.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:03.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:03.181+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:03.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:03.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:03.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:03.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:03.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:03.079+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:03.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:03.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:03.474+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:03.475+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:04.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:04.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:04.183+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:04.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:04.183+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:04.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:04.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:04.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:04.126+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:04.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:04.128+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:04.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:04.442+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:04.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:04.442+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:04.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:04.442+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:04.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:04.443+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:05.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:05.099+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:05.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:05.099+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:05.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:05.099+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:05.406 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:05.100+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:05.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:05.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:05.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:05.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:05.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:05.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:05.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:05.170+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:05.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:05.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:05.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:05.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:05.427+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:06.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:06.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:06.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:06.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:06.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:06.148+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:06.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:06.149+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:06.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:06.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:06.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:06.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:06.423 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:06.118+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:06.423 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:06.118+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:06.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:06.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:06.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:06.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:06.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:06.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:06.420+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:07.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:07.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:07.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:07.172+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:07.173+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:07.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:07.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:07.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:07.123+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:07.400 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:07.123+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:07.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:07.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:07.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:07.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:07.397+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:08.086 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:07.830+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35073 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:08.192+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:08.192+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:08.192+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:08.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:08.192+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:08.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:08.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:08.084+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:08.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:08.085+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:08.697 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:08.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:08.698 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:08.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:08.698 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:08.432+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:08.698 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:08.432+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:09.400 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:09.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:09.400 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:09.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:09.400 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:09.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:09.400 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:09.151+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:09.401 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:09.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:09.401 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:09.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:09.401 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:09.091+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:09.401 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:09.092+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:09.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:09.398+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:09.398+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:09.398+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:09.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:09.399+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:10.380 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:10.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:10.380 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:10.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:10.380 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:10.118+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:10.380 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:10.119+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:10.381 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:10.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:10.381 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:10.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:10.381 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:10.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:10.381 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:10.097+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:10.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:10.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:10.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:10.378+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:10.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:10.379+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:11.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:11.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:11.052+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:11.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:11.053+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:11.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:11.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:11.144+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:11.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:11.145+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:11.761 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:11.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:11.762 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:11.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:11.762 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:11.426+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:11.762 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:11.426+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:12.381 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:12.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:12.381 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:12.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:12.381 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:12.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:12.381 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:12.154+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:12.381 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:12.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:12.382 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:12.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:12.382 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:12.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:12.382 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:12.082+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:12.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:12.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:12.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:12.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:12.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:12.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:12.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:12.380+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:13.151 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:12.834+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35170 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:13.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:13.037+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:13.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:13.037+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:13.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:13.037+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:13.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:13.038+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:13.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:13.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:13.341+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:13.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:13.342+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:13.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:13.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:13.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:13.150+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:14.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:14.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:14.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:14.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:14.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:14.045+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:14.342 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:14.046+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:14.342 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:14.128+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:14.343 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:14.128+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:14.343 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:14.128+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:14.343 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:14.129+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:14.622 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:14.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:14.622 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:14.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:14.622 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:14.340+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:14.622 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:14.340+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:15.326 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:15.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:15.326 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:15.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:15.326 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:15.169+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:15.327 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:15.170+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:15.327 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:15.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:15.327 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:15.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:15.327 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:15.067+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:15.327 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:15.068+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:15.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:15.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:15.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:15.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:15.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:15.324+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:15.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:15.325+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:16.315+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:16.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:16.316+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:16.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:16.316+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:16.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:16.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:16.213+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:16.214+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:16.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:16.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:16.097+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:16.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:16.098+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:17.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:17.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:17.298+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:17.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:17.299+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:17.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:17.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:17.208+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:17.209+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:17.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:17.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:17.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:17.061+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:17.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:17.062+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:18.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:17.839+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35267 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:18.070+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:18.070+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:18.070+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:18.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:18.070+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:18.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:18.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:18.277+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:18.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:18.278+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:18.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:18.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:18.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:18.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:18.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:18.229+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:18.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:18.230+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:19.307+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:19.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:19.308+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:19.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:19.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:19.078+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:19.078+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:19.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:19.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:19.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:19.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:19.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:19.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:20.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:20.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:20.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:20.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:20.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:20.048+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:20.329 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:20.048+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:20.329 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:20.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:20.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:20.268+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:20.330 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:20.269+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:20.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:20.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:20.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:20.326+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:20.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:20.327+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:21.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:21.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:21.291 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:21.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:21.291 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:21.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:21.291 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:21.220+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:21.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:21.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:21.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:21.033+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:21.292 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:21.034+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:21.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:21.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:21.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:21.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:21.288+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:21.289+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:22.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:22.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:22.266+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:22.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:22.267+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:22.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:22.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:22.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:22.036+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:22.037+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:22.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:22.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:22.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:22.196+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:22.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:22.197+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:23.162 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:22.844+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35363 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:23.163 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:23.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:23.163 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:23.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:23.163 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:23.047+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:23.163 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:23.048+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:23.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:23.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:23.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:23.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:23.302+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:23.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:23.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:23.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:23.160+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:23.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:23.161+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:24.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:24.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:24.074+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:24.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:24.075+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:24.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:24.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:24.254+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:24.255+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:24.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:24.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:24.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:24.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:24.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:24.139+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:24.641 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:25.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:25.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:25.269+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:25.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:25.270+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:25.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:25.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:25.115+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:25.116+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:25.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:25.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:25.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:25.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:25.428 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:25.167+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:26.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:26.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:26.270+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:26.270+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:26.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:26.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:26.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:26.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:26.181+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:26.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:26.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:26.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:26.103+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:26.428 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:26.103+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:27.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:27.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:27.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:27.314+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:27.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:27.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:27.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:27.137+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:27.138+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:27.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:27.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:27.100+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:27.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:27.101+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:27.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:27.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:27.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:27.708 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:02:27.709 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-19T07:02:27.709 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-19T07:02:27.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:27.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:28.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:27.848+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35459 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:28.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:28.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:28.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:28.094+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:28.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:28.095+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:28.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:28.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:28.081+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:28.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:28.082+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:28.343+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:28.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:28.344+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:29.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:29.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:29.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:29.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:29.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:29.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:29.392 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:29.128+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:29.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:29.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:29.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:29.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:29.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:29.116+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:29.393 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:29.117+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:29.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:29.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:29.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:29.390+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:29.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:29.391+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:30.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:30.091+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:30.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:30.091+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:30.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:30.091+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:30.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:30.092+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:30.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:30.073+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:30.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:30.073+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:30.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:30.073+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:30.422 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:30.074+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:30.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:30.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:30.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:30.419+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:30.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:30.420+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:30.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:31.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:31.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:31.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:31.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:31.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:31.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:31.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:31.124+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:31.043+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:31.426 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:31.043+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:31.043+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:31.427 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:31.044+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:31.763 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:31.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:31.763 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:31.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:31.764 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:31.436+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:31.764 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:31.436+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:32.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:32.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:32.398 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:32.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:32.173+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:32.174+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.023+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:32.399 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.023+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:32.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:32.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:32.396+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:32.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:32.397+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:33.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:32.853+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35555 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:33.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:33.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:33.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:33.155 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:32.989+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:33.415 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:33.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:33.415 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:33.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:33.415 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:33.153+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:33.415 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:33.154+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:33.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:33.413+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:33.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:33.413+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:33.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:33.413+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:33.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:33.414+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:34.403 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:33.948+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:34.403 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:33.948+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:33.948+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:33.948+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:34.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:34.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:34.117+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:34.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:34.118+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:34.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:34.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:34.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:34.401+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:34.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:34.402+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:35.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:35.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:35.121+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:35.122+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:34.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:35.371 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:34.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:35.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:34.975+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:35.372 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:34.975+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:35.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:35.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:35.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:35.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:35.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:35.369+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:35.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:35.371+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:36.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:36.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:36.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:36.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:36.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:36.150+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:36.352 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:36.151+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:36.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:36.019+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:36.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:36.019+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:36.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:36.019+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:36.353 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:36.020+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:36.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:36.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:36.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:36.351+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:36.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:36.351+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:37.314 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.013+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:37.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:37.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:37.315 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:37.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:37.316 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:37.117+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:37.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:37.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:37.313+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:37.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:37.314+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:38.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:37.859+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35653 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:38.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:38.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:38.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:38.108+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:38.108+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:38.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:37.980+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:38.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:38.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:38.302+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:38.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:38.303+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:39.263 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:39.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:39.264 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:39.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:39.264 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:39.140+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:39.264 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:39.141+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:39.264 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:38.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:39.265 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:38.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:39.265 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:38.995+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:39.265 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:38.996+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:39.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:39.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:39.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:39.262+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:39.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:39.262+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:39.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:39.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:40.234 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:39.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:39.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:39.972+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:39.972+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:40.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:40.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:40.235 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:40.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:40.236 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:40.185+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:40.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:40.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:40.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:40.310+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:40.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:40.311+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:41.314 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:41.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:41.314 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:41.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:41.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:41.138+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:40.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:40.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:40.989+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:41.315 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:40.989+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:41.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:41.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:41.312+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:41.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:41.313+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:42.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:41.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:42.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:41.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:42.145 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:41.980+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:42.145 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:41.981+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:42.282+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:42.282+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:42.282+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:42.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:42.282+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:42.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:42.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:42.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:42.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:42.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:42.143+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:42.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:43.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:42.862+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35748 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:43.021+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:43.021+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:43.021+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:43.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:43.022+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:43.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:43.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:43.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:43.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:43.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:43.180+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:43.661 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:43.180+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:43.661 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:43.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:43.662 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:43.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:43.662 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:43.299+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:43.662 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:43.300+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:44.258 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:44.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:44.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:44.181+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:44.181+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:44.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:44.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:44.259 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:44.000+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:44.260 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:44.001+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:44.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:44.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:44.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:44.256+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:44.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:44.257+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:45.302 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:45.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:45.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:45.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:45.205+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:45.303 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.012+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:45.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.012+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:45.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:45.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:45.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:45.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:45.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:45.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:46.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:46.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:46.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:46.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:46.303 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:46.197+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:46.304 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:46.197+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:46.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:46.304 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:46.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.964+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:46.305 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:45.965+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:46.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:46.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:46.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:46.300+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:46.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:46.301+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:47.255 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:47.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:47.255 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:47.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:47.170+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:47.171+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:46.926+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:47.256 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:46.927+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:47.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:47.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:47.253+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:47.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:47.254+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:48.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:47.867+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35845 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:48.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:47.883+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:48.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:47.883+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:48.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:47.883+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:48.132 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:47.884+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:48.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:48.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:48.239+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:48.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:48.240+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:48.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:48.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:48.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:48.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:48.131+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:49.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:48.869+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:49.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:48.869+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:49.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:48.869+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:49.127 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:48.869+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:49.222+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:49.222+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:49.222+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:49.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:49.223+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:49.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:49.125+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:49.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:49.126+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:50.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:49.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:50.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:49.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:50.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:49.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:50.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:49.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:50.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:50.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:50.240+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:50.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:50.241+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:50.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:50.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:50.142+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:50.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:50.143+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:51.161 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:50.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:51.161 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:50.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:51.162 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:50.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:51.162 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:50.809+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:51.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:51.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:51.159+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:51.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:51.160+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:51.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:51.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:51.290+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:51.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:51.291+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:52.129 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:51.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:52.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:51.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:52.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:51.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:52.130 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:51.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:52.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:52.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:52.321+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:52.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:52.322+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:52.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:52.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:52.127+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:52.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:52.128+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:53.143 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:52.871+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 35941 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:53.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:52.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:52.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:52.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.144 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:52.880+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:53.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:53.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:53.320+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:53.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:53.321+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:53.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:53.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:53.138+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:53.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:53.139+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:53.896+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:53.896+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:53.896+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:53.897+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:54.370 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:54.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:54.370 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:54.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:54.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:54.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:54.371 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:54.113+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:54.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:54.368+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:54.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:54.368+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:54.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:54.368+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:54.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:54.369+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:54.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:02:54.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:02:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:55.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:55.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:55.106+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:55.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:55.107+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:54.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:54.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:54.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:55.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:54.879+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:55.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:55.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:55.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:55.372+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:55.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:55.373+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:56.145 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:55.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:56.145 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:55.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:56.145 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:55.875+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:56.146 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:55.877+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:56.352+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:56.353+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:56.353+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:56.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:56.354+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:56.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:56.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:56.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:56.143+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:56.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:56.144+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:57.138 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:56.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:57.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:56.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:57.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:56.856+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:57.139 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:56.857+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:57.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:57.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:57.379+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:57.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:57.380+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:57.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:57.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:57.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:57.136+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:57.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:57.137+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:57.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:02:58.106 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:57.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:58.106 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:57.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:58.106 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:57.838+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:58.107 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:57.839+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:58.107 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:02:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:02:57.876+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36037 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:02:58.361 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:58.104+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:58.362 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:58.104+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:58.362 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:58.104+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:58.362 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:58.105+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:58.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:58.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:58.359+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:58.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:58.361+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:59.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:58.870+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:59.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:58.870+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:59.086 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:58.870+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:02:59.087 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:58.870+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:59.401 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:59.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:59.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:59.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:59.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:59.084+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:02:59.402 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:02:59.085+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:02:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:59.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:59.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:59.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:02:59.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:02:59.400+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:59.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:59.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:59.911+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:00.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:02:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:02:59.912+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:00.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:00.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:00.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:00.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:00.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:00.111+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:00.177 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:00.112+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:00.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:00.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:00.425+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:00.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:00.425+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:00.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:00.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:00.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:00.988 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:03:00.988 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-19T07:03:00.991 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124/remote/smithi134/crash 2024-01-19T07:03:00.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/crash -- . 2024-01-19T07:03:01.015 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:00.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:01.015 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:00.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:01.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:00.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:01.016 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:00.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:01.019 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/573cd99a-b692-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124/remote/smithi150/crash 2024-01-19T07:03:01.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:01.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:01.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:01.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:01.404 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:01.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:01.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:01.122+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:01.405 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:01.123+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:01.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:01.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:01.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:01.402+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:01.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:01.403+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:02.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:01.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:02.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:01.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:02.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:01.863+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:02.168 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:01.864+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:02.165+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:02.165+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:02.165+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:02.167+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:02.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:02.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:02.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:02.404+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:02.427 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:02.404+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:03.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:02.881+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36131 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:02.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:02.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:02.892+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:02.892+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:03.662 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:03.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:03.662 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:03.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:03.662 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:03.200+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:03.663 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:03.200+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:03.663 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:03.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:03.663 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:03.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:03.663 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:03.394+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:03.663 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:03.395+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:03.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:03.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:03.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:03.865+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:03.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:03.866+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:04.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:04.345+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:04.345+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:04.345+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:04.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:04.346+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:04.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:04.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:04.198+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:04.199+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:04.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:04.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:04.910+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:04.911+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:05.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:05.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:05.388+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:05.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:05.389+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:05.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:05.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:05.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:05.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:05.176+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:06.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:05.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:05.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:05.887+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:06.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:05.888+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:06.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:06.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:06.211+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:06.212+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:06.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:06.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:06.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:06.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:06.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:06.399+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:06.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:06.399+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:06.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:06.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:06.893+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:06.894+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:07.429+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:07.429+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:07.429+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:07.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:07.429+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:07.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:07.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:07.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:07.201+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:07.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:07.201+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:08.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:07.885+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36228 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:07.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:07.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:07.876+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:08.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:07.877+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:08.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:08.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:08.438+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:08.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:08.439+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:08.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:08.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:08.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:08.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:08.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:08.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:08.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:08.190+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:09.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:08.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:09.166 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:08.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:09.166 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:08.834+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:09.166 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:08.835+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:09.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:09.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:09.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:09.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:09.164+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:09.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:09.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:09.803 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:09.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:09.803 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:09.435+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:09.803 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:09.436+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:09.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:09.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:09.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:10.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:09.801+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:10.472 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:10.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:10.473 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:10.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:10.473 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:10.199+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:10.473 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:10.199+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:10.783 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:10.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:10.783 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:10.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:10.783 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:10.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:10.783 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:10.471+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:11.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:10.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:11.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:10.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:11.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:10.781+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:11.160 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:10.782+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:11.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:11.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:11.158+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:11.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:11.158+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:11.766 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:11.446+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:11.766 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:11.446+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:11.766 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:11.446+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:11.766 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:11.447+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:12.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:11.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:12.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:11.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:12.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:11.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:12.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:11.788+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:12.491 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:12.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:12.492 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:12.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:12.492 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:12.190+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:12.492 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:12.191+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:12.836 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:12.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:12.836 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:12.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:12.836 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:12.489+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:12.836 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:12.491+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:13.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:13.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:13.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:12.890+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36324 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:12.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:12.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:12.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:13.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:12.832+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:13.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:13.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:13.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:13.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:13.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:13.226+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:13.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:13.227+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:13.839 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:13.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:13.839 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:13.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:13.839 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:13.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:13.839 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:13.535+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:13.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:13.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:13.837+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:14.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:13.837+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:14.537 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:14.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:14.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:14.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:14.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:14.216+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:14.539 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:14.217+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:14.822 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:14.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:14.822 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:14.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:14.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:14.535+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:14.823 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:14.536+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:15.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:14.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:15.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:14.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:15.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:14.820+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:15.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:14.821+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:15.504 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:15.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:15.504 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:15.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:15.505 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:15.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:15.505 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:15.251+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:15.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:15.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:15.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:15.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:15.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:15.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:15.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:15.503+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:16.063 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:16.063 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:16.063 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:15.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:16.063 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:15.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:16.160 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:16.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:16.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:16.219+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:16.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:16.219+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:16.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:16.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:16.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:16.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:16.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:16.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:16.748 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:16.455+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:16.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:16.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:16.745+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:17.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:16.747+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:17.488 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:17.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:17.489 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:17.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:17.489 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:17.218+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:17.489 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:17.219+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:17.759 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:17.487+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:17.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:17.487+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:17.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:17.487+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:17.760 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:17.488+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:18.020 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:17.895+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36422 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:18.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:17.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:18.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:17.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:18.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:17.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:18.020 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:17.758+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:18.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:18.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:18.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:18.252+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:18.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:18.252+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:18.806 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:18.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:18.806 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:18.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:18.806 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:18.510+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:18.806 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:18.511+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:18.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:19.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:18.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:18.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:18.804+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:19.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:19.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:19.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:19.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:19.483+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:19.484+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:19.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:19.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:19.282+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:19.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:19.282+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:19.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:19.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:19.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:19.763+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:20.503 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:20.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:20.504 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:20.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:20.504 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:20.288+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:20.504 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:20.289+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:20.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:20.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:20.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:20.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:20.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:20.502+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:20.802 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:20.502+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:21.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:20.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:20.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:20.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:20.801+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:21.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:21.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:21.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:21.471+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:21.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:21.472+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:21.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:21.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:21.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:21.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:21.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:21.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:21.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:21.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:22.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:21.822+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:22.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:22.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:22.423+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:22.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:22.424+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:22.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:22.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:22.257+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:22.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:22.258+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:23.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:22.900+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36517 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:22.812+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:22.812+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:22.812+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:22.813+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:23.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:23.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:23.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:23.292+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:23.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:23.292+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:23.664 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:23.292+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:23.664 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:23.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:23.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:23.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:23.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:23.392+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:23.665 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:23.393+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:23.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:23.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:23.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:23.799+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:23.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:23.799+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:24.255+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:24.255+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:24.255+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:24.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:24.256+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:24.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:24.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:24.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:24.414+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:24.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:24.415+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:25.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:24.817+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:24.817+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:24.817+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:25.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:24.817+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:25.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:25.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:25.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:25.285+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:25.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:25.286+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:25.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:25.445+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:25.445+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:25.445+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:25.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:25.446+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:25.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:25.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:25.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:26.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:25.826+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:26.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:26.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:26.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:26.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:26.307+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:26.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:26.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:26.434+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:26.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:26.435+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:26.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:26.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:26.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:26.837+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:27.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:27.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:27.293+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:27.293+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:27.458+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:27.458+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:27.458+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:27.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:27.459+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:28.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:28.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:28.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:27.904+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36614 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:27.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:27.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:27.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:27.836+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:28.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:28.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:28.337+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:28.338+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:28.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:28.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:28.452+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:28.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:28.453+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:28.818+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:28.818+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:28.818+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:28.819+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:29.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:29.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:29.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:29.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:29.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:29.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:29.456 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:29.329+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:29.835 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:29.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:29.835 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:29.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:29.835 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:29.454+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:29.836 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:29.454+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:29.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:29.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:29.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:29.834+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:30.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:30.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:30.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:30.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:30.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:30.299+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:30.501 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:30.300+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:30.791 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:30.498+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:30.792 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:30.498+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:30.792 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:30.498+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:30.792 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:30.500+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:30.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:30.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:31.232 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:31.515 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:31.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:31.516 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:31.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:31.516 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:31.251+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:31.516 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:31.252+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:31.767 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:31.514+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:31.767 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:31.514+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:31.767 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:31.514+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:31.767 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:31.514+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:32.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:31.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:31.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:31.801+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:31.802+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:32.559 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:32.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:32.559 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:32.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:32.559 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:32.245+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:32.559 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:32.246+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:32.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:32.557+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:32.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:32.557+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:32.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:32.557+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:32.810 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:32.558+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:33.176 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:32.909+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36709 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:32.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:32.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:32.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:32.808+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:33.569 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:33.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:33.569 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:33.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:33.569 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:33.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:33.569 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:33.273+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:33.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:33.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:33.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:33.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:33.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:33.568+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:33.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:33.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:33.809+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:33.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:33.810+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:34.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:34.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:34.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:34.300 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:03:34.300 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-19T07:03:34.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-19T07:03:34.332 INFO:teuthology.orchestra.run.smithi134.stderr:grep: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph.log: No such file or directory 2024-01-19T07:03:34.334 INFO:tasks.cephadm:Compressing logs... 2024-01-19T07:03:34.334 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-19T07:03:34.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:34.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:34.403 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-19T07:03:34.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-19T07:03:34.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.chvejw.log 2024-01-19T07:03:34.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.higalz.log 2024-01-19T07:03:34.404 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.chvejw.log: gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mon.smithi134.log 2024-01-19T07:03:34.404 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.higalz.log: gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mgr.smithi134.qrxokr.log 2024-01-19T07:03:34.406 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mon.smithi134.log: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mgr.smithi134.qrxokr.log: gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.1.log 2024-01-19T07:03:34.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.0.log 2024-01-19T07:03:34.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.2.log 2024-01-19T07:03:34.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-volume.log 2024-01-19T07:03:34.409 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-19T07:03:34.423 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-volume.log.gz 2024-01-19T07:03:34.430 INFO:teuthology.orchestra.run.smithi134.stderr: 95.9% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.chvejw.log.gz 2024-01-19T07:03:34.438 INFO:teuthology.orchestra.run.smithi134.stderr: 95.1% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mds.cephfs.smithi134.higalz.log.gz 2024-01-19T07:03:34.610 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:34.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:34.610 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:34.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:34.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:34.250+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:34.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:34.250+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:34.608+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:34.608+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:34.608+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:34.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:34.609+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:34.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:34.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:34.797+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:34.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:34.798+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:35.289 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mgr.smithi134.qrxokr.log: file size changed while zipping 2024-01-19T07:03:35.289 INFO:teuthology.orchestra.run.smithi134.stderr: 90.0% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mgr.smithi134.qrxokr.log.gz 2024-01-19T07:03:35.570 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mon.smithi134.log: file size changed while zipping 2024-01-19T07:03:35.570 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-mon.smithi134.log.gz 2024-01-19T07:03:35.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:35.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:35.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:35.290+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:35.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:35.291+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:35.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:35.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:35.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:35.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:35.633+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:35.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:35.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:35.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:35.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:35.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:35.804+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:35.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:35.806+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:36.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:36.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:36.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:36.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:36.245+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:36.680+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:36.680+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:36.680+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:36.682+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:37.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:36.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:36.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:36.821+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:36.822+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:37.440 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:37.665 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:37.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:37.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:37.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:37.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:37.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:37.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:37.265+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:37.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:37.913+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36807 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:37.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:37.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:37.790+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:37.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:37.791+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:37.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:37.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:37.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:37.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:37.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:37.665+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:38.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:38.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:38.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:38.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:38.635 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:38.227+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:38.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:38.229+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:38.633+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:38.633+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:38.633+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:38.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:38.634+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:38.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:38.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:38.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:38.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:38.804+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:39.417 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-01-19T07:03:39.417 INFO:teuthology.orchestra.run.smithi134.stderr: 95.0% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.2.log.gz 2024-01-19T07:03:39.625 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-01-19T07:03:39.626 INFO:teuthology.orchestra.run.smithi134.stderr: 95.0% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.0.log.gz 2024-01-19T07:03:39.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:39.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:39.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:39.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:39.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:39.272+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:39.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:39.273+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:39.751 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-01-19T07:03:39.751 INFO:teuthology.orchestra.run.smithi134.stderr: 94.9% -- replaced with /var/log/ceph/573cd99a-b692-11ee-95ae-87774f69a715/ceph-osd.1.log.gz 2024-01-19T07:03:39.752 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-19T07:03:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m5.365s 2024-01-19T07:03:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m17.263s 2024-01-19T07:03:39.753 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.400s 2024-01-19T07:03:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:39.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:39.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:39.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:39.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:39.836+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:39.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:39.668+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:39.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:39.668+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:39.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:39.668+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:39.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:39 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:39.670+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:40.638 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:40.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:40.638 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:40.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:40.638 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:40.313+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:40.638 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:40.313+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:40.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:40.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:40.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:40.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:40.657+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:40.658+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:40.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:40.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:40.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:40.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:40 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:40.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:41.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:41.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:41.322+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:41.653 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:41.322+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:41.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:41.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:41.833+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:41.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:41.834+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:41.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:41.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:41.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:41.651+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:41.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:41 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:41.652+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:42.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:42.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:42.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:42.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:42.642 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:42.283+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:42.643 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:42.284+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:42.920 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:42.640+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:42.920 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:42.640+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:42.920 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:42.640+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:42.921 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:42.641+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:42.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:42.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:42.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:42.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:42.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:42.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:42.921 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:42.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:43.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:42 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:42.918+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36900 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:43.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:43.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:43.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:43.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:43.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:43.264+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:43.598 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:43.264+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:43.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:43.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:43.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:43.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:43.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:43.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:43.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:43.597+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:43.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:43.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:43.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:43.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:43.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:43 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:43.840+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:44.619 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:44.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:44.619 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:44.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:44.619 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:44.311+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:44.620 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:44.312+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:44.617+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:44.617+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:44.617+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:44.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:44.618+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:44.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:44.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:44.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:44.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:44 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:44.848+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:45.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:45.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:45.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:45.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:45.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:45.329+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:45.625 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:45.330+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:45.925 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:45.623+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:45.623+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:45.623+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:45.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:45.623+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:45.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:45.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:45.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:45.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:45.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:45 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:45.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:46.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:03:46.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:03:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:46.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:46.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:46.375+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:46.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:46.376+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:46.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:46.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:46.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:46.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:46.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:46.596+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:46.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:46.597+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:47.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:46.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:47.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:46.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:47.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:46.792+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:47.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:46 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:46.792+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:47.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:47.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:47.384+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:47.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:47.385+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:47.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:47.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:47.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:47.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:47.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:47.563+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:47.677 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:47.563+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:48.048 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:47.922+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 36997 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:48.049 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:47.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:48.049 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:47.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:48.049 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:47.787+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:48.049 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:47 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:47.787+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:48.554+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:48.554+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:48.554+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:48.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:48.555+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:48.388+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:48.388+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:48.388+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:48.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:48.389+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:49.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:48.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:49.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:48.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:49.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:48.808+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:49.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:48 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:48.809+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:49.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:03:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:49.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:49.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:49.567+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:49.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:49.567+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:49.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:49.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:49.409+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:49.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:49.410+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:49.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:49.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:49.836+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:49 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:49.837+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:50.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:50.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:50.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:50.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:50.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:50.372+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:50.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:50.373+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:50.609+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:50.609+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:50.609+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:50.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:50.610+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:50.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:50.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:50.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:50.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:50.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:50 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:50.832+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:51.598+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:51.598+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:51.598+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:51.676 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:51.599+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:51.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:51.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:51.379+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:51.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:51.380+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:52.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:51.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:51.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:51.852+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:51 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:51.853+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:52.633 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:52.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:52.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:52.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:52.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:52.340+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:52.634 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:52.340+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:52.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:52.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:52.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:52.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:52.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:52.632+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:52.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:52.633+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:52.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:52.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:52.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:52.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:52.840+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:53.304 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:52 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:52.928+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 37093 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:53.665 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:53.665 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:53.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:53.302+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:53.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:53.303+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:53.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:53.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:53.839+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:53.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:53.840+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:53.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:53.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:53.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:53.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:53.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:53.663+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:53.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:53 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:53.664+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:54.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:54.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:54.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:54.291+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:54.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:54.292+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:54.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:54.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:54.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:55.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:54.794+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:55.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:54.710+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:55.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:54.710+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:55.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:54.710+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:55.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:54 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:54.711+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:55.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:55.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:55.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:55.265+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:55.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:55.266+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:55.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:55.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:55.842+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:56.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:55.843+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:56.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:55.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:55.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:55.751+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:56.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:55 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:55.752+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:56.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:56.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:56.242+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:56.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:56.242+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:56.778+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:56.778+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:56.778+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:57.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:56.778+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:56.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:56.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:56.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:57.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:56 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:56.831+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:57.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:57.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:57.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:57.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:57.233+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:58.057 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:03:57.932+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 37189 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:03:58.058 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:57.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:58.058 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:57.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:58.058 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:57.747+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:58.058 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:57.748+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:58.058 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:57.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:58.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:57.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:58.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:57.830+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:58.059 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:57 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:57.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:58.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:58.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:58.233+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:58.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:58.234+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:58.698+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:58.698+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:58.698+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:58.699+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:59.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:58.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:58.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:58.789+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:58 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:58.790+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:59.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:59.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:59.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:59.231+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:03:59.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:03:59.232+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:59.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:59.664+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:59.664+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.929 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:59.664+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:03:59.929 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:03:59.665+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:03:59.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:59.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:59.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.931 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:59.788+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:03:59.931 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:03:59 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:03:59.788+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:00.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:00.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:00.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:00.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:00.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:00.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:00.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:00.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:00.688+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:00.689+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:01.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:00.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:00.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:00.829+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:00 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:00.830+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:01.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:04:01.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:04:01.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:01.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:01.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:01.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:01.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:01.187+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:01.671 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:01.187+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:01.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:01.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:01.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:01.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:01.810+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:01.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:01.669+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:01.669+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:01.669+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:01.928 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:01 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:01.670+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:02.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:02.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:02.175+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:02.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:02.176+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:02.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:02.637+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:02.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:02.637+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:02.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:02.637+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:02.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:02.638+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:02.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:02.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:02.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:02.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:02.858+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:03.227 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:02 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:04:02.937+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 37284 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:04:03.666 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:03.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:03.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:03.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:03.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:03.225+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:03.667 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:03.226+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:03.679+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:03.679+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:03.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:03.679+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:03.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:03.680+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:03.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:03.846+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:03.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:03.846+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:03.928 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:03.846+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:03.929 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:03 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:03.847+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:04.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:04:04.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:04.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:04.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:04.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:04.191+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:04.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:04.192+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:04.677+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:04.677+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:04.677+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:05.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:04.678+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:05.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:04.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:04.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:04.822+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:05.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:04 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:04.823+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:05.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:05.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:05.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:05.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:05.195+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:05.708+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:05.708+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:05.708+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:06.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:05.709+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:06.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:05.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:06.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:05.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:06.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:05.861+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:06.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:05 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:05.861+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:06.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:06.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:06.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:06.194+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:06.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:06.194+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:06.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:06.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:06.696+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:06.696+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:07.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:06.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:06.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:06.897+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:06 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:06.897+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:07.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:04:07.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:04:07.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:04:07.644 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:04:07.645 INFO:tasks.cephadm:Removing cluster... 2024-01-19T07:04:07.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 573cd99a-b692-11ee-95ae-87774f69a715 --force 2024-01-19T07:04:07.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:07.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:07.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:07.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:07.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:07.184+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:07.670 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:07.185+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:07.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:07.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:07.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:07.726+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:07.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:07.727+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:07.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:07.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:07.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:07.866+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:07.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:07.866+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:08.224 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:08.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:08.224 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:08.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:08.224 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:08.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:08.225 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:08.156+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:08.225 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:07 smithi134 systemd[1]: Stopping Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T07:04:08.225 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:07 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:04:07.943+0000 7f2e6acdc700 -1 mon.smithi134@0(probing) e2 get_health_metrics reporting 37381 slow ops, oldest is log(1 entries from seq 279 at 2024-01-19T06:26:40.592507+0000) 2024-01-19T07:04:08.225 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:04:08.223+0000 7f2e6dce2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi134 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T07:04:08.225 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134[28279]: 2024-01-19T07:04:08.223+0000 7f2e6dce2700 -1 mon.smithi134@0(probing) e2 *** Got Signal Terminated *** 2024-01-19T07:04:08.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:08 smithi134 podman[84178]: 2024-01-19 07:04:08.619929501 +0000 UTC m=+0.516150448 container died 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , RELEASE=quincy-6441305) 2024-01-19T07:04:08.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:08.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:08.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:08.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:08.926 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:08.730+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:08.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:08.730+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:08.848+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:08.927 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:08 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:08.849+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:09.384 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 podman[84178]: 2024-01-19 07:04:09.148835635 +0000 UTC m=+1.045056585 container cleanup 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, org.label-schema.build-date=20240102, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-19T07:04:09.385 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 bash[84178]: ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134 2024-01-19T07:04:09.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:09.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:09.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:09.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:09.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:09.145+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:09.385 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:09.145+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:09.652 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 podman[84192]: 2024-01-19 07:04:09.383687744 +0000 UTC m=+0.758415362 container remove 328c6cb50ef7496e84b3f2be3c9796a67e4c7f136794205eb3895d0a91135cf5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-mon-smithi134, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-19T07:04:09.652 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service: Deactivated successfully. 2024-01-19T07:04:09.653 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 systemd[1]: Stopped Ceph mon.smithi134 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T07:04:09.653 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Jan 19 07:04:09 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@mon.smithi134.service: Consumed 24.107s CPU time. 2024-01-19T07:04:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:09.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:09.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:09.862+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:09.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:09.862+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:09.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:09.681+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:09.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:09.681+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:09.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:09.681+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:09.927 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:09 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:09.682+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:10.129+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:10.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:10.130+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:10.848 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:10.693+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:10.848 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:10.693+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:10.849 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:10.693+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:10.849 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:10.693+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:11.106 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:10.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.107 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:10.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.107 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:10.868+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.107 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:10 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:10.869+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:11.358 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:11.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:11.358 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:11.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:11.358 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:11.110+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:11.358 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:11.111+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:11.973 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:11.719+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:11.973 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:11.719+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:11.973 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:11.719+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:11.973 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:11.720+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:11.974 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:11.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.974 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:11.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.974 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:11.853+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:11.974 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:11 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:11.853+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:12.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:12.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:12.407 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:12.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:12.408 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:12.152+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:12.408 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:12.153+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:13.157 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:12.763+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:13.157 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:12.763+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:13.157 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:12.763+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:13.157 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:12.764+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:13.157 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:12.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:13.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:12.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:13.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:12.840+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:13.158 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:12 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:12.841+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:13.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:13.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:13.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:13.155+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:13.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:13.156+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:14.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:13.796+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:14.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:13.796+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:14.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:13.796+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:14.119 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:13.797+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:14.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:13.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:14.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:13.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:14.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:13.825+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:14.120 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:13 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:13.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:14.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:14.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:14.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:14.116+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:14.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:14.117+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:15.141 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:14.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:15.141 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:14.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:15.141 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:14.846+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:15.141 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:14.847+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:15.142 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:14.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:15.142 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:14.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:15.142 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:14.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:15.142 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:14 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:14.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:15.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:15.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:15.139+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:15.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:15.140+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:16.029 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:15.858+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:16.029 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:15.858+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:16.029 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:15.858+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:16.030 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:15.858+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:16.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:15.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:16.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:15.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:16.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:15.835+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:16.031 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:15 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:15.836+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:16.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:16.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:16.421 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:16.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:16.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:16.171+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:16.422 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:16.172+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:17.016 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:16.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:17.016 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:16.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:16.894+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:16.895+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:16.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:16.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:16.879+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:17.017 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:16 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:16.879+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:17.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:17.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:17.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:17.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:17.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:17.215+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:17.325 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:17.216+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:18.000 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:17.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:18.000 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:17.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:18.000 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:17.904+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:18.000 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:17.904+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:18.001 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:17.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:18.001 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:17.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:18.001 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:17.850+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:18.001 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:17 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:17.850+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:18.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:17 smithi134 systemd[1]: Stopping Ceph osd.0 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T07:04:18.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.287+0000 7fecda180700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T07:04:18.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.287+0000 7fecda180700 -1 osd.0 46 *** Got signal Terminated *** 2024-01-19T07:04:18.289 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.287+0000 7fecda180700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-19T07:04:18.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:18.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:18.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:18.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:18.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:18.204+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:18.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:18.205+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.938+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:19.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:18.939+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:18.828+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:18.828+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:18.828+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:19.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:18 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:18.829+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:19.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:19.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:19.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:19.202+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:19.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:19.203+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:20.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:19.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:19.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:19.794+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:20.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:19.795+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:19.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:20.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:19.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:20.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:19.916+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:20.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:19 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:19.916+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:20.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:20.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:20.193+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:20.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:20.194+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:21.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:20.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:20.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:20.793+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:21.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:20.794+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:20.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:21.176 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:20.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:20.875+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:21.177 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:20 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:20.875+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:21.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:21.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:21.188+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:21.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:21.189+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:22.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:21.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:22.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:21.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:22.165 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:21.800+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:22.166 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:21.801+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:22.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:21.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:22.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:21.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:22.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:21.866+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:22.166 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:21 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:21.867+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:22.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:22.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:22.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:22.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:22.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:22.163+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:22.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:22.164+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:23.151 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:22.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:22.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:22.883+0000 7fecd7f81700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:36.742525+0000 (oldest deadline 2024-01-19T06:58:56.742525+0000) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0[46883]: 2024-01-19T07:04:22.884+0000 7fecd7f81700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2104 3.1f 3:fc4d2eb4:::100000031fa.00000000:head [write 0~3732 in=3732b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:22.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:22.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:23.152 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:22.831+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:23.153 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:22 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:22.831+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:23.425 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:23.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:23.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:23.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:23.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:23.149+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:23.426 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:23.150+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:24.072 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:23 smithi134 podman[85019]: 2024-01-19 07:04:23.757352278 +0000 UTC m=+5.617891141 container died c42a610a3422aba0492912f3bf7b2b272aeccbb75b409b3e4192378f92c2ddce (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-01-19T07:04:24.073 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:23.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:24.073 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:23.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:24.073 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:23.803+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:24.073 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:23 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:23.803+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:24.381 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:24 smithi134 podman[85019]: 2024-01-19 07:04:24.170766273 +0000 UTC m=+6.031305134 container cleanup c42a610a3422aba0492912f3bf7b2b272aeccbb75b409b3e4192378f92c2ddce (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-01-19T07:04:24.382 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:24 smithi134 bash[85019]: ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0 2024-01-19T07:04:24.382 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:24.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:24.382 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:24.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:24.382 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:24.167+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:24.382 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:24.168+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:24.675 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:24 smithi134 podman[85040]: 2024-01-19 07:04:24.380601537 +0000 UTC m=+0.622005087 container remove c42a610a3422aba0492912f3bf7b2b272aeccbb75b409b3e4192378f92c2ddce (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux ) 2024-01-19T07:04:25.077 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:24.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.077 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:24.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.077 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:24.758+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.078 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:24 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:24.759+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:25.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:24 smithi134 podman[85128]: 2024-01-19 07:04:24.701592588 +0000 UTC m=+0.019640889 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d 2024-01-19T07:04:25.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:24 smithi134 podman[85128]: 2024-01-19 07:04:24.831637678 +0000 UTC m=+0.149685971 container create c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240102, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3) 2024-01-19T07:04:25.078 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:25 smithi134 podman[85128]: 2024-01-19 07:04:25.076237009 +0000 UTC m=+0.394285318 container init c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_CLEAN=True) 2024-01-19T07:04:25.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:25 smithi134 podman[85128]: 2024-01-19 07:04:25.079883667 +0000 UTC m=+0.397931967 container start c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=quincy-6441305, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102) 2024-01-19T07:04:25.372 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:25 smithi134 podman[85128]: 2024-01-19 07:04:25.184556161 +0000 UTC m=+0.502604464 container attach c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-6441305, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240102, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-01-19T07:04:25.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:25.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:25.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:25.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:25.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:25.217+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:25.373 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:25.218+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:25.642 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:25 smithi134 podman[85128]: 2024-01-19 07:04:25.371836723 +0000 UTC m=+0.689885026 container died c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, org.label-schema.build-date=20240102, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-19T07:04:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:25.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:25.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:25.733+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:25.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:25 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:25.734+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:26 smithi134 podman[85154]: 2024-01-19 07:04:26.005414448 +0000 UTC m=+0.627627528 container remove c4fbb3fadecb556fa18b32896b125cbb7a68b8f6f90c7d903844a83a4af295a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-0-deactivate, RELEASE=quincy-6441305, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240102, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.31.3) 2024-01-19T07:04:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:26 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.0.service: Deactivated successfully. 2024-01-19T07:04:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:26 smithi134 systemd[1]: Stopped Ceph osd.0 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T07:04:26.426 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jan 19 07:04:26 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.0.service: Consumed 18min 16.789s CPU time. 2024-01-19T07:04:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.224+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.225+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:26.427 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 systemd[1]: Stopping Ceph osd.1 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T07:04:26.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.499+0000 7fc89542b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T07:04:26.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.499+0000 7fc89542b700 -1 osd.1 46 *** Got signal Terminated *** 2024-01-19T07:04:26.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:26.499+0000 7fc89542b700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-19T07:04:27.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:26.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:26.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:26.763+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:27.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:26 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:26.764+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:27.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:27.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:27.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:27.244+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:27.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:27.245+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:28.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:27.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:27.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:27.810+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:28.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:27 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:27.810+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:28.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:28.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:28.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:28.275+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:28.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:28.276+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:29.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:28.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:28.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:28.858+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:29.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:28 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:28.859+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:29.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:29.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:29.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:29.306+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:29.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:29.306+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:30.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:29.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:29.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:29.847+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:30.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:29 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:29.848+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:30.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:30.260+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:30.260+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:30.260+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:30.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:30.261+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:31.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:30.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:30.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:30.824+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:31.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:30 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:30.825+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:31.675 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:31.303+0000 7fc891a29700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:23.135906+0000 (oldest deadline 2024-01-19T06:58:43.135906+0000) 2024-01-19T07:04:31.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1[54532]: 2024-01-19T07:04:31.304+0000 7fc891a29700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2091 3.19 3:9fbc213f:::100000031f1.00000000:head [write 0~858 in=858b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:32.176 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:32 smithi134 podman[85242]: 2024-01-19 07:04:32.015453313 +0000 UTC m=+5.642701915 container died 980268e5252b2e460b26629b30082b98227c2bdf48ffcb7cb0371af1f2b49156 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, org.label-schema.build-date=20240102, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-01-19T07:04:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:31.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:31.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:31.867+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:31 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:31.868+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:32 smithi134 podman[85242]: 2024-01-19 07:04:32.417305856 +0000 UTC m=+6.044554449 container cleanup 980268e5252b2e460b26629b30082b98227c2bdf48ffcb7cb0371af1f2b49156 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1, org.label-schema.build-date=20240102, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3) 2024-01-19T07:04:32.669 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:32 smithi134 bash[85242]: ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1 2024-01-19T07:04:32.926 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:32 smithi134 podman[85264]: 2024-01-19 07:04:32.668750488 +0000 UTC m=+0.649106060 container remove 980268e5252b2e460b26629b30082b98227c2bdf48ffcb7cb0371af1f2b49156 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240102, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, RELEASE=quincy-6441305, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-01-19T07:04:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:32.885+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:32.885+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:32.885+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:32.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:32 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:32.886+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:33.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:32 smithi134 podman[85356]: 2024-01-19 07:04:32.901047548 +0000 UTC m=+0.019654145 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d 2024-01-19T07:04:33.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:33 smithi134 podman[85356]: 2024-01-19 07:04:33.028128402 +0000 UTC m=+0.146734991 container create 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=quincy-6441305, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.31.3) 2024-01-19T07:04:33.290 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:33 smithi134 podman[85356]: 2024-01-19 07:04:33.289242881 +0000 UTC m=+0.407849473 container init 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240102) 2024-01-19T07:04:33.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:33 smithi134 podman[85356]: 2024-01-19 07:04:33.292525696 +0000 UTC m=+0.411132288 container start 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-6441305, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-01-19T07:04:33.611 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:33 smithi134 podman[85356]: 2024-01-19 07:04:33.389172856 +0000 UTC m=+0.507779454 container attach 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-6441305, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-01-19T07:04:33.884 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:33 smithi134 podman[85356]: 2024-01-19 07:04:33.610798264 +0000 UTC m=+0.729404861 container died 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240102, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-19T07:04:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:33.882+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:33.882+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:33.882+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:34.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:33 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:33.883+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:34.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 systemd[1]: Stopping Ceph osd.2 for 573cd99a-b692-11ee-95ae-87774f69a715... 2024-01-19T07:04:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:34 smithi134 podman[85382]: 2024-01-19 07:04:34.326633521 +0000 UTC m=+0.710243977 container remove 08934f0aefe8ef5880a8392f8800b5ab4231061564b1866bae8d15af384318e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-1-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240102, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, RELEASE=quincy-6441305, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-01-19T07:04:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:34 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.1.service: Deactivated successfully. 2024-01-19T07:04:34.676 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:34 smithi134 systemd[1]: Stopped Ceph osd.1 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T07:04:34.677 INFO:journalctl@ceph.osd.1.smithi134.stdout:Jan 19 07:04:34 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.1.service: Consumed 18min 21.515s CPU time. 2024-01-19T07:04:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.820+0000 7f975030c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T07:04:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.820+0000 7f975030c700 -1 osd.2 46 *** Got signal Terminated *** 2024-01-19T07:04:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.820+0000 7f975030c700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-19T07:04:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:35.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:35.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.860+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:35.177 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:34 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:34.861+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:35.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:35.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:35.907+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:36.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:35 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:35.907+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:37.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:36.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:37.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:36.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:37.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:36.878+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:37.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:36 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:36.878+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:38.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:37.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:37.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:37.913+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:38.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:37 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:37.914+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:39.175 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:38.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6804 osd.3 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:38.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6812 osd.4 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:38.891+0000 7f974a906700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.150:6820 osd.5 ever on either front or back, first ping sent 2024-01-19T06:58:29.050152+0000 (oldest deadline 2024-01-19T06:58:49.050152+0000) 2024-01-19T07:04:39.176 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:38 smithi134 ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2[62179]: 2024-01-19T07:04:38.891+0000 7f974a906700 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24305.0:2095 3.8 3:14a61925:::10000002de7.00000000:head [write 0~2317 in=2317b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-01-19T07:04:40.546 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:40 smithi134 podman[85460]: 2024-01-19 07:04:40.257483969 +0000 UTC m=+5.564631509 container died cbc144916de2dee1f88011db3d4580f07096f7ab49e89158f3c13b61d9a5a750 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-6441305, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240102, org.label-schema.license=GPLv2) 2024-01-19T07:04:40.547 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:40 smithi134 podman[85460]: 2024-01-19 07:04:40.545915126 +0000 UTC m=+5.853062665 container cleanup cbc144916de2dee1f88011db3d4580f07096f7ab49e89158f3c13b61d9a5a750 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-6441305, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240102, CEPH_POINT_RELEASE=) 2024-01-19T07:04:40.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:40 smithi134 bash[85460]: ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2 2024-01-19T07:04:40.926 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:40 smithi134 podman[85481]: 2024-01-19 07:04:40.774293472 +0000 UTC m=+0.519370202 container remove cbc144916de2dee1f88011db3d4580f07096f7ab49e89158f3c13b61d9a5a750 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, RELEASE=quincy-6441305, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS) 2024-01-19T07:04:41.501 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.108660352 +0000 UTC m=+0.017402937 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d 2024-01-19T07:04:41.501 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.206477827 +0000 UTC m=+0.115220417 container create 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-6441305, ceph=True, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240102, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-19T07:04:41.502 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.417111168 +0000 UTC m=+0.325853764 container init 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6441305, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240102, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-19T07:04:41.502 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.420338718 +0000 UTC m=+0.329081302 container start 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, RELEASE=quincy-6441305, org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-01-19T07:04:41.502 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.50035172 +0000 UTC m=+0.409094322 container attach 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240102, RELEASE=quincy-6441305, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-01-19T07:04:41.925 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:41 smithi134 podman[85570]: 2024-01-19 07:04:41.693113679 +0000 UTC m=+0.601856262 container died 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, ceph=True, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240102, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-6441305) 2024-01-19T07:04:42.274 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:42 smithi134 podman[85596]: 2024-01-19 07:04:42.270263355 +0000 UTC m=+0.571891023 container remove 2e2bf03811e942e30eb14a963230f25fd93f70591664dc263eddc8ebb3e7f3fe (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-573cd99a-b692-11ee-95ae-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-6441305, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240102) 2024-01-19T07:04:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:42 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.2.service: Deactivated successfully. 2024-01-19T07:04:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:42 smithi134 systemd[1]: Stopped Ceph osd.2 for 573cd99a-b692-11ee-95ae-87774f69a715. 2024-01-19T07:04:42.676 INFO:journalctl@ceph.osd.2.smithi134.stdout:Jan 19 07:04:42 smithi134 systemd[1]: ceph-573cd99a-b692-11ee-95ae-87774f69a715@osd.2.service: Consumed 18min 23.788s CPU time. 2024-01-19T07:04:50.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:04:50.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:04:53.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:02.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:02.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:05.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:17.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:17.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:20.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:23.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:23.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:23.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:23.644 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:05:23.644 INFO:tasks.cephadm:Teardown complete 2024-01-19T07:05:23.645 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1591, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1845, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1848, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 1724, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 180, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephadm.py", line 348, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:05:23.665 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-19T07:05:23.676 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-01-19T07:05:23.691 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-19T07:05:23.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-19T07:05:23.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:23.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:26.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:35.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:35.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:38.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:50.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:50.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:53.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:56.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:56.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:56.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:05:56.988 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:05:57.043 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:05:57.061 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-19T07:05:57.071 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-19T07:05:57.072 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T07:05:57.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:05:57.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:05:57.094 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-19T07:05:57.099 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T07:05:57.099 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-19T07:05:57.100 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 26 +785us[ +785us] +/- 93ms 2024-01-19T07:05:57.100 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -538us[ -528us] +/- 77ms 2024-01-19T07:05:57.100 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -152us[ -152us] +/- 72ms 2024-01-19T07:05:57.100 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T07:06:00.156 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:09.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:09.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:06:12.220 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:24.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:24.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:06:27.292 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:30.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:30.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:06:30.368 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:30.368 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:06:30.369 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:06:30.385 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-19T07:06:30.396 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-19T07:06:30.397 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-19T07:06:30.415 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-19T07:06:30.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:30.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:06:33.567 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:42.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:42.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:06:45.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:06:57.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:06:57.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:00.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:03.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:03.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:03.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:03.772 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:07:03.773 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:07:03.780 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-19T07:07:03.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-19T07:07:03.802 INFO:teuthology.task.internal:Duration was 3376.838098 seconds 2024-01-19T07:07:03.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-19T07:07:03.814 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-19T07:07:03.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-19T07:07:03.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:03.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:03.867 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T07:07:06.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:15.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:15.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:18.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:30.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:30.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:34.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:37.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:37.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:37.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:37.084 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:07:37.085 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:07:37.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-19T07:07:37.116 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-19T07:07:37.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-19T07:07:37.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:37.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:40.224 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:07:49.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:07:49.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:07:52.284 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:04.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:04.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:07.360 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:10.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:10.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:10.428 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:10.428 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:08:10.429 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:08:10.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-19T07:08:10.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-19T07:08:10.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:10.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:10.496 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-01-19T07:08:13.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:22.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:22.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:25.600 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:37.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:37.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:40.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:43.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:43.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:43.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:43.740 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:08:43.741 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:08:43.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-19T07:08:43.769 INFO:teuthology.task.internal:Transferring archived files... 2024-01-19T07:08:43.770 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124/remote/smithi134 2024-01-19T07:08:43.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-19T07:08:43.813 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124/remote/smithi150 2024-01-19T07:08:43.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:43.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:46.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:08:55.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:08:55.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:08:58.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:10.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:10.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:14.012 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:17.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:17.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:17.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:17.084 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:09:17.085 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:09:17.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-19T07:09:17.114 INFO:teuthology.task.internal:Not uploading archives. 2024-01-19T07:09:17.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-19T07:09:17.125 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-19T07:09:17.126 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-19T07:09:17.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:17.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:17.148 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 19 07:05 /home/ubuntu/cephtest 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 19 07:08 /home/ubuntu/cephtest/archive 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 19 06:10 /home/ubuntu/cephtest/archive/syslog 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 265292 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Jan 19 06:10 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 265291 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Jan 19 06:10 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 786433 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 19 07:06 /home/ubuntu/cephtest/archive/audit 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 786435 172 -rw------- 1 ubuntu ubuntu 173317 Jan 19 07:06 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-01-19T07:09:17.149 INFO:teuthology.orchestra.run.smithi134.stdout: 265330 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Jan 19 06:16 /home/ubuntu/cephtest/cephadm 2024-01-19T07:09:17.150 INFO:teuthology.orchestra.run.smithi134.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-01-19T07:09:20.195 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:29.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:29.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:32.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:44.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:44.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:47.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:50.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-01-19T07:09:50.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T07:09:50.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-01-19T07:09:50.396 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-01-19T07:09:50.397 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi150 2024-01-19T07:09:50.412 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-19T07:09:50.425 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-19T07:09:50.439 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-01-19T07:09:50.440 INFO:teuthology.nuke:Checking targets against current locks 2024-01-19T07:09:50.467 DEBUG:teuthology.nuke:shortname: smithi134 2024-01-19T07:09:50.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T07:09:50.490 DEBUG:teuthology.nuke:shortname: smithi150 2024-01-19T07:09:50.490 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T07:09:50.493 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 06:01:29.156475', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ9tsHnbIlHfuRN24Az3gKwr6oYVI2AMLDFh8/ngO0GhHek8AUVOOJoIjszuTsRS09+3q3OyocQXSNQa5BFHzpc='} 2024-01-19T07:09:50.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7521124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 06:01:29.158719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJVfPBi7HetMwVhieXrb1/31pCfddChakSk8rx6raRJkUU5FqQajV20To/4zLM/P1zNPT0yK2+Vr/DVkpjt3zSo='} 2024-01-19T07:09:50.539 INFO:teuthology.orchestra.console.smithi134:Power off 2024-01-19T07:09:50.540 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-19T07:09:50.566 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-01-19T07:09:50.567 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:50.585 INFO:teuthology.orchestra.console.smithi150:Power off 2024-01-19T07:09:50.585 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-19T07:09:50.596 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-19T07:09:50.608 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-01-19T07:09:50.608 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:50.630 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-01-19T07:09:54.597 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:54.631 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:54.707 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-19T07:09:54.742 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-01-19T07:09:58.708 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:58.744 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:09:58.819 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-19T07:09:58.855 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-01-19T07:10:02.824 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:10:02.856 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-19T07:10:02.934 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-01-19T07:10:02.966 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-01-19T07:10:03.036 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-01-19T07:10:03.067 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-01-19T07:10:03.173 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3376.838098049164 failure_reason: 'Command failed on smithi134 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 573cd99a-b692-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cc05a3c497cb461fa448f060d4858251 status: fail success: false 2024-01-19T07:10:03.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T07:10:03.250 INFO:teuthology.run:FAIL